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/PREFACE 



This report As one of several that is being written a3 part of Abf 
Associates* broader sttidy of the National Institute of Education's Research 
and Development Utilization (RDU) fcfogram. The report has a relatively * 
narrow focus: the protest by which pool? of products (referred to in this, 
report as the * knowledge base") were developed to serve- 'clients! needs, the 
typei of, product^- that we^e included for dissemination, and the types that 
were chosen by sites. Other reports wiil analyze other aspects of the prograc 
the problem-solying process that schools uWerwent in selecting a product to 
meet 'the it neejis, the management of thV^Jpul projects and the linkage systems 
that were established* tjie training and support of the. linking agents, the 

role of NIB in designing and monitoring the 5DU program, and the impacts of 

1 * * 

the program at tfte site* level, ' 1 . • * ^ 

The RDU Program 

In June of 1976, the National Institute of Education (KIE) established 
the Research aod Development Utilization program as a new dissemination 
effort to help schools clarify and solve local problems. The RDU program 
emphasized a. research-based) rational approach bo \ocal school improvement 
^through the use of existing, validated products of federally funded research 
and development activities. This program was designed to achieve three 
major ob}ectives; , / 

. , ■ 

• to help sdjools alleviate 'specific, locally defined problems 
< in the areas of basic skills and 'career education; 

• 'to helj> school and community personnel learn about the 
"products of educational research "and development: and 

• to increase unSerstanding pfc how the local program improvement 
process can be better managed and become more effective. 

The RDU program is unusual among federally funded dissemination strate- 

gies because it is equally concerned with the dissemination and use of R&D . 

products $r>d *ith the development of local organizational capabilities to 

solve problems. Other federal programs have tended to concentrate on either 
• ■ ' • , -* 

product dissemination 'or local capacity building. 



The RDU Strategy * " * 

*The core of the RDU approach to provide e&ch participating site with 
assistance in following the sequence of agtivrties listed below; ^ m 

■* * r 

** « t 

» identification of a problem or set of problems; A 

• examination of alternative solutions to the problem, focusing 
' ' particularly on the products "of educational R&D; 

• selection of a specific solution considered to be appropriate 
to alleviate the problem; ■ 

» ■ ' implementation of the solution; 'and, 

•* evaluation arid incorporation of both the solution, and th£~ 
0 problem- solving process • ^ ' « 

The service delivery \system of the RDU pr%g'ram operated through seven 
projects, each o^ which coordinated a network of organizations and. indivi- 
duals mvqlved^m the provision of services and information to local school 
districts. As a whole, the seVen projects operated in 2> states and served 
over 300 schools or school districts over a" three-year period (JL976-1979) . 

"linking agent* tin many cases, former teachers or school administrators)/ 
assigned by the project, coordinated!, the sjgrrvices provided to the local 

schools and school districts and helped guide the local school personnel in a 

■ — ^ « 
school improvement process. Each project supported two or more linkers but 

their roles varied among projects,- Most^operated out of an ^intermediate 

service agency, o£ a^state education 'agency,, and each servedyt specific set 

of local schools or school districts.. The range of a linker's possible roles. 

included facilitating the decision-making proqess by clarifying goals and 

providing leadership, and mediating among autonomous and sometimes competing" 

organizations whose resources 'and services needed toJae coordinated. _ T^e 

seven projects wer& regionally distributed an3 involved the following." 

The Northwest Reading "Consortium (NRC), under the overall direction of 
the Washington State Education Agency-, operated as a consortium of four 
states in the Northwest; Washington, Qr«goa, Alaska, and Idaho, The 'project 
built upon thp existing Right to Read program in the *f oujr states* (The^ Right 
to Read Program is a nationwide program sponsored. by the U.S. Office of 
Education to eliminate functional illiteracy. ) The Northwest Regional^ 
Educational Laboratory- was also an affiliate, providing training to ptoject 
staff and support in the development of a pool of RSDJ products. 
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The Georgia State Department of Education operated a project which 
provided funds and services* to all participating school districts located in 
three Cooperative Educational Service Areas. The emphasis of the BDU project 
in Georgia was on building local school district -capacities in the early 
stages of planning and program selection. The implementation phase of the * 
problem-solving model was subsequently-carried out with federal funds avail- 
able through the state .department of education under Title IV-C of the 

Elementary and Secondary Education Act and with other state funds. 
• * - 

• jfre- ^ennsylvania Department of Education developed and coordinated 
9 a school improvement process which involved the participation and , resources 
of several organizations: Research for Better Schools (a regional education 
lab), Research and Information Services for Education <(a state-wide informa- 
tion and dissemination service), the Learning Research and Development Center 
at the University of Pittsburgh, and the state's Intermediate Units. The 
project's problem-solving nodil, "which was developed by the participating 
resource agencies, involved numerous defined 3teps, including a series of 
formal training sessions in problem solving at the school sites. * 

The National Education Association (NEA) operated its project in collabora- N 
. tion-with the state education Jagencies and corresponding state education 
associations in 12 states: Alabama, California, Iowa, Massachusetts, Michigan/ 
Minnesota, Ohio, Pennsylvania, Tennessee, Washington, Wisconsin, and Wyoming* 
In contrast to the other RDU projects, "this project focused exclusively on 
thfc improvement of teacher insexvice education. Services were provided by 
two liftking agents in each state who trained local staff. 

• * / , 

The Flo* ida Department of Education served as prime contractor in a 
linkage system which also involved the state universities (especially Florida 
State ahd the University of Florida), and eight of the state's Teacher 
Education Centers (TECs) . An important feature of this project is that 
training in group problem- solving techniques was provided not only to the 
linking agent (one of whom wa? located in each TEC), but also to selected 
local school' staff. The school site facilitators, with'tfte he'lp of the 
linking agents, were responsible for lead i - aq t-j ie staff at their sites 
through the entire problem-solving process.,^ 

The Career Education Dissemination Project of the Michigan Department of 
Education was designed to help local sites meet the requirements of state 
career education legislation passed in 1974. One of. the project's major 
objectives w$s to develop a permanent dissemination and diffusion System Xn ^ 
career education. Because of this emphasis on permanence, the project 

^attempted to work with existing structures and personnel in the st'ate's 
Intermediate School Districts rather than build new or-es. The primary 

Ifetrategy was to provide direct training and programmatic funds to coordinators* 
who*were staff metobe'rs at local 1 sites > 
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' ' The NETWORK f a noo-prpfit research'and servic* organization in Andover, 
Mass* coordinated a consortium of agencies, in six states: in Minnesota, the 
agency involved was a teacher center associated with a university ; m in Washing- 
ton, a local school district; in California, a regional education laboratory 
sponsored by trfE; in Kansas, an independent state-wide education diffusion 
organization? in Connecticut, a cooperative service agency supported by 
local schooi districts; artf in Massachusetts, a division of The NETWORK 
itself. This "project was formed mainly* to* improve the utilization of RiD 
prompts in reading in selected local schools. The linking ^gents provided 
assistance to the local sites, while a considerable amount of technical 
assistance and support was provided to the linking agents By the project 
off ice .in Andover* • ■ 

Seme common features ran throughout, the seven projects and in the 
structure o? support services provided to local schools 



• ' the operational a project headguaptess to coordinate the * / 

services. supplied to schools; 

• the development and administration of a knowledge bas,e 
composed of educational research and development products; 

• tlie development of training and technical assistance * 
components to serve the project 'S. linking agents and/or 
school* staff; and V t 

- • the development of projeqt evaluation and research activities ' # 

to facilitate local self-evaluation by t{ie schdol sites. 
w § 

The Study of the RDU Program 

soy^mber 1977 AbtVssodiates Inc., a social science research firm 
based ir, Cambridge, Massachusetts, w^s contracted to conduct a study of the 
&DU program. The study addresses six major issues: 

• how relatipnships are managed between various agencies which . 
have tfie expertise and resources to help local schools solve 

„ problems; * ^ 1 

• ^tP wfiat degree an intervention programNsuch as RDU can *help 

schools overcome barriers to successful problem solving 
(such as limited access' to information or ^lack of planning, * 
skills, etc.) ; , 

» to what degree the products of educational R&D*are ^relevant 
to the problems and contexts of local schools? 

» what the impact is of the products of educational R&D once 
they hAve been adopted and impleMfited; f 

» what factors contribute to deinstitutionalization of the 
RDU approach within a vaciet^f .organizations; and 

p how linking agents coordinate the flow of ektfernal resources 
to schools, and whether this hel^f the schools solve problems 
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• ♦ * 
All of the seven projedts have completed the federally supported service 

delivery plan of their activities. However, the research effort^ by Abt 
Associates will continue unfcl^early in' 1981. The lessons that can be learned 
free the activities and outcomes -of the^ program will^have ijoportant impli- 
cations for the design and operation of dissemination programs in education/ 
as well as for the design and management of future federal, state, and local 
efforts to impr ov^schools . 

We would like to acknowledge the assistance of our colleagues on, the 
RDU study, who h^ve provided us with insights* data, and jiioral support. 
These include: , Sheila Rosenblura, Kent Chabotar , Diane Kell, Greg Spencer, 
4 JimMolitor, and Thea Moskat. Earlier drafts of this report also benefited 
fra£ comments made by Nancy Ames, Robert Dentler, and Sara D. Sieber. NIB 
staff members, including our pJoject officer r 3phn Egerroeier, Michael Kane, 
Ward Mason, rfnd Naida Bagenstos, also contributed to improvements in both our 
interpretations and writing. Special thanks go to Thea Moskat and Mary-Ellen 
Perry foe tflfeir unfailing attention to the details or producing and editing 
this report, 



< 



v 



"J 

10 



t t 



\ ■ 



CHAPTER I ' 
. INTPpDDCTION ' 

* 

The RDU Program 4 # . \ 

The R&D Utilization (RDU) program was designed by. NIE as an "action/ 

research" program, aimed primarily at improving local school practices, # 

Improved practice was supposed to result from a number of' initiatives , 

* * 
including; . , 

• the application of R&D products or ideas to school * 
'problems; " 

9 the- development of a problem-solving process , whereby, 
schools would systematically identity^ such problems 
and select and implement new ideas; and 

• the organization of a linkage system' , whereby national, f 
state /and other external, resources would be made 

available to school personnel, ^ > 

This raultifaceted approach was an "action* program in that actual changes^ , ( 

were attempted in a variety of schools; in the eftd, over 300 schools ^or 

school districts participated in the program, organized according to s ev en j 

project awards ; four were made to state departments of education (Florida,* 

Pennsylvania, Michigan, and Georgia), two were made to organizations represent- t 

ing agencies across the country (the National Education Association, and a 

non-profit educational organization, The NETWORK) , and one was \niide to a 

regional consortium administered by the state of Washington (Northwest 

Reading Consbrt ium--NRC) . The approach was a "research" program in that the 

ensuing activities were exceptionally well -documented by the projects them- . 

selves and have been the subject of subsequent -study by an independent 

contractor, Abt Associates Inc. (see Chabotar and Kell, 1978; and Louis ef 

s 

al., 1979), 

♦ # The three initiatives mentioned above were maintained in the organization 

of each of the seven projects- The present report focuses on' only-the first s 

of these three initiatives to determine how different, types of educational 

products or ideas were selected and adopted by schools. The other two 

initiatives, involving the problem-solving process and linkage systems , 

wiil be treated in separate reports, and the final" report* will integrate 

• • + 

the findings fraa all three initiatives.* 



* These reports will be completed during 198p and 1981, 
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The Role 'of t^he RDU Program within the Broader NIE Mandate 

The use of .flew ideas to address local school problems was a part of the 
RDU»program that was well-suited to NIE's broader mission (Raisin, 1979): 

0 T 

"The Institute, , ,will have principal responsibility fc for dissemi- 
nation of the results of research ♦ ♦ ♦ (T]he Institute should also 
seek to stimulate the demand of state and local agencies tot, , 
educational improvement and* to strengthen their capacity for-' 
implementing ^ested innovations: (House Report 92-554, 1971) 

Thus, in designing the RDU program, one of.NIE's aims was to show that exist- 
ing products and ideas in education could, in fact, be useful to schobls* 

The design, of the RDU program underwent many modifications and much re- 
view before it was finally described in a Request for Proposals (RFP) (see 
Corwin, forthcoming, '1980) » The RFP was issued in 1976, and it requested 

- ■ * 

interested parties to propose action projects, ^each of which would enlist" 
a large number of local schools to test the existing products or* ideas in 
actual practice. ' Although there was* not necessarily clear consensus among 
NIE staff regarding the detailed p&iorities for the RDU program (s*e Corwin, 
forthcoming, 1980), the RFP did nevertheless represent the official solici- 
tation 'to which bidders responded. Thus, it is important to note how the 
RFP defined the .use to be made^of educational ^pro^ucts- or ideas. First, the 
major stated purpose was to support projects that would; 

♦.♦help to meet (the NIE requirement of helping to solve educa- 
tional problems in schools] by providing services to schools to 
implement and use existing research and development outcomes* 
(Emphasis added, J (NIE, 19 75 s : 1 1 

Similarly, T the solicitation gave four criteria for assessing the long-£fcrra 

effects of the RDU program,^ and all were relate^ to the use of educational 

products, or "outcomes,"* Acceptable effects were defined as: 

• an increase in knowledge (by educational. i * 
personnel) of the "nature and utility R&D 
outcaaep"; * 

£ • an increase in the "number 7 of formal assessments * J , 

• of the 'potential of specific R&D outcomes for, 
local use"; m ■ * 

• an increase in," instances where R&D outcomes or • 
appropriate adaptations are implemented success- ♦ * 
fully"? or * ' ^ 

• * a reduction of "local problems as a result of 
r the successful implementation of R&D outages 

or appropriate adaptations to meet* local n&ds.*" + • 
(NIE, 1975:1) 



* The terms "products," "idea," and "outcome" are used synonyinously in this 
© report < M 
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« * * * 

The U3c Of Educational Products- in the RDU Program 

" 1 * ' « • 

* As priglnally planned/ this report wafs intended to examine, the degree to 

whicti* schools adopted aiyd implemented existing educational products as a 

result of their involveinent in the RDU. i«&gran. However, as we began to 

examine the question £f-» adopt Ions , it'SXue clear that this simple statement* 

of the issue to be Examined, ignored enormous efforts, made to define what was 

•an^el-igible pfbdu'ct, bot^j'by.the seven projects and at ttfe ^school level- t 

Thus, -this report wil*l not examine th§> implementation process, nor will 

it discus* the outctees of product * implementation by schools. These issues % 
' , • ' . * - , » 

will be discussed in Gur final report Preliminary results are also pre- 

* t * * * t 

sented -,n Louiiv't 1980J / ' . y , 

. # -The RFP jyrovided'-Only limited guidance for determining a product's 
elijguJt>*jfit^# t An existing R&D product; was assumed by the -original RF? to - 
include "curricula, produces, skills, programs, ms£runeft£s, teaching and 
ayinagement methods an^ techniques , ajid the like" (NIE, 1975:2). ht first 



glance , such products appear not to be especially different from thos 
emanating from what Lindblom and £ohen (1979) call "ordinary" knowledge. t 
However, the' p?P went farther stipulated th^t acceptable products had to 

^tt^sfy two cri teria ^ evidenceffi g|^f f ectiveness and evidence of transport- 
abi lity f*om one site to anofcheK^pTlE, 1975:12). The RFPv howgvtfr, £id not 
^rdvi^any guidance concerning the ways' m which a project could test -for 
tivese criteria.. * The projects could, for instance, assume sucH evidence 

' X * " / ' ' ' ^ " 

existed if a produqp v $Ame from an acceptable source (e.g., the NIE catalog of 
products, 1975); or the projects could develop their own review or validation 
procedure. Whatever method,was used, thfe lefck of specificity led to' varia- 
tions*^ project procedures tha£ will be aj^j^ssed irTchapter II* 

Thus> throughout 'the early portions of the RDU program, a ^reat deal 
ol attention was given ^to the development**©^ a Knowledge base that contained^ 
acceptable products** and the process by which each school ultimately selected 
a product for adoption, t A knowledge base, which refers to a specific col- 
. iection'of educational products, 'was developed for" each of the" seven RDU 

grojjects to serve the needs* of the schools or school districts within the „ 

' " * * 

project* By design, the seven knowledge bases covered different curriculum^ 

• 

♦•What constituted an acceptable product changed somewhat during the life 
'of the RDU progr*jj| Thi^ is discussed in greater detail on page 22. 



topics/ reflecting^he priorities of each project. Thus, basic skills 
(reading and math) were the dominant topics covered by the knowledge bases 
in Florida, NRC, Pennsylvania, and The NETWORK." Hie Georgiar knowledge base 
covered these topics but also included -career education and^dis^rict-level 
-planning; the Michigan project focused mainly on career education; and the 
NBA proj^Cty focused mainly on inservice training (which^in turn covered a 
large variety of specific curriculum topics)* 

* Because, the RFP required that knowledge-base -products b^ validated, a f 
major issue for each of the project* had to do. with" a product's acceptability 
by evaluative standards ^nd the desire 'to have schools adopt only products 
that had been previously validated through empirical testing* Thus, the 
original intent was to assure some. quality control over the .new ideas that 
would be implemented within any' given school. Our analysis, however, suggests 

^thafc-man^y schools were eventually presented with numerous nonvalidated 

products for adopt ion, .i ncluding, the following examples: , - 

• Two projects. {Georgia and Michigan) > provided assist$hce on 

a curriculum topic — career education^-in which the bulk of 1 
the existing products* had not been validated. As n result, 
only 20, percent of one project's products^(Miehigan) were 
were beli-eved to have been validated. Similarly, one. of- - 

* * the most popular products i-n Georgia ( # "It ttorks") ,\which f 

helpedVa^jchopl to develop a management system for safeer 
- ~educaty^!||M^K*t^^^ * • 



One o^f^l^fflKro^ects (Georgia) also pr6vlded assistance on 
♦a dj^strlct^ttanagement topic--planning-«in which roost of £he 
available products we^fe also nonvalidated* * 

In two other projects (Florida and NRC) , individual ^schools 
were reported to have adopted products that had not been 
validated. - - 1 * 

In another project (NBA) ,* the major topic of assistance— 
inservice training— covered so many curriculum areas that 
little attention could be given to whether the identified 
products had previously been validated. Based *on later 
analysis by the project's 1 staff, many had not been. J 

-In only two projects (Pennsylvania arid The NETffORKfc did it 
appear that schools had adopted validated products rflone. 



• • * • 

In summary, the knowledge # and products beihg offered to schools had not 

■ , «» 

necessarily passed through an empirically validated screening process. The 

products might, however, have served other school needs, including genuine 
inprovements in school practice.* The point of this report is to show that, 
regardless oTthese effects, much effort was expended on the problem of.^ 
•what to adopt, - and yet a nixed pattern of adoptions occurred. In thne 
remainder of this report, which is based on data collected through field 
interviews in the fall of 1978 and spring of 1979, as well as on the analysis 
of project documents, we analyze how the projects and schools dealt with 
the adoption problem. The following chapters fogus on two major activities^ 
• that affected eventual adoption patterns! the development of the knowledge 
base {Chapter II), and the matching of available products to an individual 
school's needs (Chapter; III) . These chapters attempt to indicate the dif- 
ficulties encountered in identifying and screening acceptable products. 
The final chapter (IV) discusses the research and policy implicat ions 



of, thestf* findings. 



J 



J 



* The actual implementation effects will be reported in the final report, 
which -integrates all the different aspects of the RDU pf^ram. 
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- ' CHAPTER II 

OiOPE RATING TgE KNOWLEDGE BASE " ' , 

# The first critical activity- conducted by each of .the seven RDIT projects 
was the organisation of the knowledge base. This chapter of the report de- 
scribes ^ 1) the organisational structure of the seven knowledge bases. 2} the 
procedures whereby existing educational products were acquired for the knowl- 
edge base, both ^initially *nd on a continuing basis? and 3} the results of 
these •procedures, in terms -o^jUffe rent types of products* 
Organisational Structure of the Knowledge Base 

Each of the sevefr;project& that was awarded a contract had a distinct 
structdre, although* each project basically followed the same product, 
oro^ess/ and linkage system blueprint. ( (See Louis, et al- # forthcoming.) 
Because of tAe .structural variations, however t ^similar project functions 
were not Iperforped by the sane type of agency ^or were the functions per- 
formed in. exactly- the sane way, a diversity that was desired in the origi- 
nal RFP {NIE, 1975:6") I * " /1 

Staffing- 3>attei*CT7and* Level of Effort . The knowledge bases of the 
sev^n projects were located in different types of agencies, were staffed 
by different tj^es of individuals, and operated under different levels 
of. effort „ThJ^£ of the prtfy^ts (Th^JETWORK, Georgia, and the NEA) 
operated their knowledge bases with their own staffs, within the project's 
host organisation. The other four had contracts with an independent lab or 
--^organi,satxoA r '-or Kith an intermediate service agency, for the operation of . 
., their knowledge base, but one of these (Michigan) also conducted significant 
in-ho use f unct ions • 

The staff ipgi and level of effort varied among th« knowledge bases (see 
Table 1). "Staffing ranged from 1.5 full- time equivalents (FY 1978) at The 
NETWORK to ^ three full-tine equivalents in Pennsylvania, and total expend!-, ^ 
" tures for operating the knowledge base (thr-cJu^h FY 1979) varied from about 
$40, 000. in Michigan to' $230,000 in the HRC. This variation may be explained 
by two factors. Firgg,, the knowledge base staffs ware formed at different 



* Staff in, the. Michigan RDU pyoject office searched for, acquired, and did 
a preliminary review of^ products. These tentatively acceptable products 
were t^eiK sent to a subcontractor-rThe Kalamazoo Valley Intermediate 
Service District-- where they were 'abstracted, evaluated, and reviewed 
for bias*. 



Table 1 

Staffing and Level of .Effort for 4 'Bach Project's^ Knowledge Base 



/ V 

Project 
(Location) 



Developer of 
Knowledge Base 
(Location) V 



Date Knowledge 
Base Staff 
Began 



Size of Knowledge 
Base Staff, FY 1978 
(t of PTEs) 



Total Khowledge 
Base .Expend ittK^s 

Throuqh FY 1979 
■ * ($00fr) 



Florida * 
(Tallahasee) 

Pennsylvania 
(Harrisburg) 



HRC< 
(Seattle) 



Options for R&D fc 
(Tallahassee) 

Research & Jnfor- 
nation Services 
for Education 

, (RISE) 
Philadelphia) 

Northwest Regional 
Educational Lab 
(Portland, OR) 



9/76 



76 



6/76 



Michigan - 
(Ea8t,£ansing) 



Georgia * 
(Atlanta) 

The NETWORK 
(Andover, MA) 

KEA 

(Washington,*, 
DC) 



Kalamazoo Vafley* 
Iritermediate 
School District, 
and in-house* v 

in-house 



in-house 

\ 

^in-houao 



11/77 
'1/77 

4 

6/76 
11/76" 



2.5 
3<~* 



1.5 



112.2 



125.3 



231.8 



37.5 



HA 



NA 



HA 



MA - not available (knowledge base expenditures cannot be distinguished from other 
project expenditures) * J 
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times? all were not assembled at the same time that the ROU projects were 
established* yi<i£hlg)|n, for example, awarded a- contract to the Kalamazoo 
Valley- ISD tfo assist .in the development of the knowledge base" in September 
1977, 15 months after, the Michigan projedt was funded {June 19/&6). The 
Georgia knowledge base also recruited its first staff member late, in January 
1977, ' The expenditures for these knowledge bases from their start through 
fisca^year 4979, therefore, were less than those for the projects t&at had 
begun earlier/ * * - [ > 

Second, some knowledge- base staffs performed more functions than others. 

' - j * I 

The knowledge-base staffs for the Michigan, NETWORK, and tfEA protects, for 

exaaple* spent ill thfcir effort on .the search for, review of, a*id acquisition 

of Rkh products. The others (Florida, Georgia, Pennsylvania, and^the NRC) , 

however, divided their tiw^ among several functions — administration, training, 
* 

technical assistance, research and evaluation — as well as on building the 

knowledge base. The knowledge- base staffs that performed these additional 

functions- -functions that were performed by staff of other project components 

\ • _ . 

in the other three projects— generally w^je larger £nd had more funds, 

■* Tr^ividuals with varying skills' and backgrounds malte up the knowledge- 
base 3tat£sV All had at least one person with specific prior experience 
in developing educational curricula, this person usually assumed the major 
/responsibility for: A ■ 

• g&theting products, relevant to sites 1 needs; 

~ t -r- 

• Assisting sites in thfe selection of a product; 
and ' 

* t 

• assuring the fidelity of the knowledge-base ,/ 

operations to overall program object ive$. 

Other persons on the knowledge-base staffs included teacher assistants and 
* / 

* descr iters' in the NEA, both* of whom were responsible for writing brief de- 

* * 

scriptions of the products in the knowledge base,* and a consultant/evalu- 
ator in Michigan, jho hafr the responsibility for eveluat itjej^he usefulness 
and assessing the validity of ea£h product that was considered for inclusion 
in the knowledge base* , 

k 

» * 

* The prodjuct descsiptions in most projects were no longer than one page. 
In the case of'the NEA, however, the descriptions were eight pages long, * 





»3fcrM types of documents were produced by the knowledge base staffs, 
* 

either for use by the staff and the linking agents* or for distribution to 
th* sites (Table 2). M.1 projects hod a coeaplete list of the products 
in the knowledge base. Florida, NRC, Georgia, and the NEA developed written 
syntheses of the products by content area or tiy *fie instructional strategy 
used, and all projects wrote brief individual descriptions of each product , 

in the knowledge base. These documents were intended*to assist the khowl- 

N 

edge-base staff, the linking agents, and the sites in selecting a product 
appropriate to sites 1 needs. (An example pf a product description is shown 
in Pigure 1*) 

Contents of the Knowledge Base , Each RDU project emphasized specific 
problem areas and acquired R&D products relevant to sites' needs in those 
areas. In its solicitation, HIE said it wanted to give^ fecial attention 
to two areas, basic skills and career education. However, these sugges- 
tions were ( riot intended to restrict offerors. >!s long as an area could be 
shown to - -be— of hi^tr pnor^t^-to local- arid state of fici^s J _ajjd_a3_lorvg as 
a "useful range of R&D resources [was] available in that problem area," 
vNIE," JL975-.8) projects were permitted to address any substantive area they,, 
chose . ' 

With the exception of the NEA project, all projects chose one or both 
o^the problem areas proposed by HIE. Pive projects chose to focus on 
problems in basic skills. Of these five, the NRC. and The NETWORK limited 
themselves to reading; Pennsylvania and Plorida chose to emphasize reading 
and foathematics; and Georgia chose to focus_on-problems in reading, mathe- 
matics, an^career education. The Michigan project decided to emphasize only 
products in cateer education * and the NEA project proposed inservice educa- 
tion as its problem area. 

As the knowledge-base staff began to communicate with the sites about 
their specific problems and needs, the content area of some of the k/iowledge 
bases broadened. The Georgia project, for example, expanded its knowledge 
base tq^include products on district-level planning. Tfc* NEA also expanded 
its knowledge base* Inservice products in special education, early child- 

t 

* The RZ}U program had various personnel, called iinkino agents, wfcoae 
explicit tasks were to coordinate the services provided to local sites, 
guide local 'school personnel through the school imp* clement process, 
and taciliate corounications between local staff and the projects. 
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Table 2 

i 

Documents Produced by the Knowledge Bases, by Project 



Project|\ 



DOCUMENT 



Complete LLsC 
of Products in 
Knowledge Base 



Product 
Descriptions 



Synthases 
of 

Product Topics 



.Maintains Cop^y 
of 

Actual. Product 



Plbr ida 

Pennsylvania 

NRC 

Michigan - 
Georgia 
The NETWORK 
NBA . 



X 
X 
X 
X 
X 
X 
X 

I 

-k- 



^ 



X 
X 
X 
X 
X 



X 
X 

X 
X 
X 
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— An' Exanjl* of a* Product Inscription, 



RE*Cr\G 



WCCUCT 



toz B*s5lnq 



CEVEiCPy 



Cx4s? -at/ Ct.-.ttr :or fred^r.? >**trdctiii v * * ., 
S*lt UXt City, I'un f — 



Otscnptoii ? t *d~nc ?ro?r&*ft 

gleneitiry Sece.id4rv Icf j;^-.; 
^I'.^a.i Instr-ct-of. 
sat Kjtre- 1 Systtr 
Znservice Titciir SdusAsssr 
frehjVLor Ch4T9i 
fteedin? Centers 



MltfT)*[* Costs 



Teener laser's::! AT.d Cofi»uliArt~- 
Jl""5 p*r d*y plus 1x7tr.su 



Ovffttf* 1 

tot*.--? &sprs'i t reid-J^ —utr^ction tforttoccts sr --J-erer i top. =4. erees— 

thro^-tout tr.e t4tion, tfte etad2 o£ tA*s * *;;r55uii*e.j $8'vo?occ* £xxr?*t 
;m:t: optrittf a readers cl*r.-z ior lUc*™? ^eipor.iee frcj Scudert* 

Kuldrti. *rn*i 44*— ti . ^ond-cts 
reteir:* at J sprteds . m .iz>rr*i-z~. X*ie 
C4*ci.ss liSTJii^-tf -std *n p ro<rria 

AT* *4*f».l±.t 13 A V4T-tty Of I 

ert* ta"! r-*l 1 end ot7*-*"*"-o ft *2 ptf 
ter** Teeo-er* 2*-* leer- to trplo, t.*t 
l«*C~»r? ^ec^iC-ies ^rd l til* use extsf 
tig reid.ro t-s ;*ro-*ce *r-s sAterieli 

Teac-ers of eie^e-tAr, at2 »e;nisr 



J Co^o*e*«*i on 



£ 

< 1 

♦ £ 



^3 Ac^: ■ v 



tC.vT .s 54 ltd 0- tie -*-4t 4 .4r;«r 

-^2+: y£ s*'jie*.t» cat 5« rt*r ^ei ty 
tri-*-"? teerteji *r ;tedi.tc. "tor; Ani 
•€<p ?4onr«r.4i •nstruot-or T*,e ■ 



04^4*,, 



^rea^tr ».7T.*?*:4T5t taat rAteris- *r. 
sfie,-t.ti ;4~ts *r, ft*d-"^ *cr.«ve2te--t 
-t*5i ~m ..t^r.-c .s * ti.cr 

-** -tr? t. fc *t --i^t ^r*^4-*t 



EvilvAiion Evidence of £ffectiverttss^ 

Dur-: r; 4n 3-acrtr, ?«r.Jd/ # tC?J ?r»^t 
>rpd»*-.* i*i /uri y; /ocas 

il'ir^ 4.M • J /t4XS , a SI 

ccr^pirti 5 And . 6 yt«r 7>ir,s r«spt;- 
*t for s^ . as re is_r^- 

Ziies-^AzZ— -* ?^ i 7 Tests 



< ^ 





Cervif* 



I^c e - ve"»l*or Rtouneir.tnu 

service td*cit.5n 
5£«;4ie s-t V-ii cemte 

;r*t.C4* b.x:di.-q st4ff tv^d«afe Aswfincei i CU<ms 

«t4l*£LT9 t>« tetnod* tr4i.im5 _ 

seti.cn 1 ot epprsxinA-ei'jf ^-CiTi ere 



for ttAc^ers 
Tr4Lii^5 11 



Co«" ftw jCh ^'y;''*^^ 



The ptetyisry Center f?r Med-r.g 
Instruction h44 conducted nunerovs 



n4cess4Tf ^or te*cn4rs co cot^.ete 4 - — — ^ *~ « ™— r« «- — ^ s ruiivo^ 

ou*tJ»t£ia *n«trtct.on*i ?*c>*9t exported Mie4ra. fc i et^dies ia cnt 4T«e rttdin? * < ^ 



tnt center* 



Sefvictl jU-Hlfiit * 
" H0fe*tsrtr9 itr/icif *re ev4ilA01e frcs 

ApO'OHli * # 

CSrt :Lis4A&A4Uor ?4view ?sne- 
«ort*ve*t ?4ii^n? Coniortii= *^**>, 



S4t4 Are eve^lAOle upoft revest. 



CV-^4 Aeidl Director 

attsi^ Rio^:»nd Drive 
S4ls EeJce Cif/. fftArf 
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hood education, and multicultural education wdre added to those inservice 
products on colics that had initi^ly been deemed relevant to the sites' 
needs.* , . * 

The number of products in the knowledge base ranged from 42* (The NETWORK) 
to 61S (KCA) * Except r for the NEA project/however, no knowledge base con- 
. twined more than 15g' products (see Table 3)» The products in the knowledge- 
bases were directed to administrators and teachers, in elementary and secondary 

\ 

schools, and to linking agents.** More products, were available fd^ elemen- 

■ v 

t«y teachers and students, however, than were available footljose Liy seCond- 

ary schools; similarly, more products were directed to teadhers than to 

administrators. In subsequent interviews, the project personnel indicated 

that th6 paucity of secondary school products was a major shortcoming among 

the available R&D products. * 

c \ 
Acquisition of Products for the Knowledge Base 

The* preceding section described the general organization, curriculum 
content., and size of the seven knowledge base?* The point of the discus- 
sion has been 'to suggest the extensive efforts made by the seven projects 
to develop an appropriate \ now ledge tfase. The key step that detemined'the 
Contents of the knowledge base, however, had to do with the procedures for 
acquiring specific educational products that were, to be^part of it, Eaclj 
project dejvelope^ different procedures, and these are of critical impor- 
tant^ in understanding the ultimate degree" £o which each knowledge base 
-actually contained acceptable products* 

Search Procedure * In looking for products for 'the knowledge base^jsacjy 
project initially examined a^arVety of sources • These sources included: 



* The NBA prpject benefited f roo a <pilot effort that had been completed 
before the PDU program began.. However,. the pilot effort was too nar- 
row, and KEA's knowledge base eventually included inservice produces 
on 14 topics. These included: Assessing student growth, career educa- 
tion, Classroom management, early childhood education, English language 
arts, individualized instruction, mathematics, motivation, multi- 
Cultural education, reading, science, sooial studies, special education, 
and teaching strategies. 

**The NEA project was the major exception. Because the HEA is a profesr 
sional organisation whose membership consists primarily of teachers, 
the products in the HEA 1 s knowledge base were limited to teachers. 
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Number of Knowledge Base Products ir* Various 
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Curriculum Areas, by Project, Spring 1^79 






• 












'CURRICULUM AREA 




T 


Total * 


Knowledge 
Base • 


1 

Basic Skills 
Reading Hath 


Career 
Education 


Inservice 
BdUuation 


District-Level 
^.Planning 


Other, 


Products 
in Knowl- 
edge Base 


• 

Florida 


68- 2t* 

n 


7 




t 


3 


§ 

92 v 


Pennsylvania 


94 18 




4 ' 

i 


■ • 


7 

i 


123 * 




59 


■ 




; * 




59 


Michigan 




,150- 








150 


-Georgia 


49 * 25 


22 








. MS 


The NETWORX 


42 






t 

— • 




42. 




— 

• • 




. 615* 


Km 

1 


* 


615 ' 


4 The 615 insorvice products included 30 on assessing student growth, 15 on 
career education , 40 on classroom management, 20 on early childhood ¥ 
education, 30 on Englieh language arts, 6S on individualized instruction, 
55»on mathematics, 70 on motivation, 25 on multi-cultural* education, 60 
on reading, 20 pn science, 45 on social studies, 40 on special education, 
and 100 on teaching strategies. 
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• existing 'catalogs of R&D products , published by the 
* -National Diffusion .Network <NDN), NIE itself , and 

* commercial publishers i\ _ 

• existing "R&D products that were already part of sotne 
state or local, information ^center; 

< 

4 other published sources/ such as pRIC; 

* h recommendations from NIE labs and centers* and 

• ■ * 

v • vord~6f-«mth recommendations by state and local per- 
sonnel/, of ten as a pfesult of coomuaications with sites 
« in^ other regions of the country, ■ 

The projects varied m the extensivenesp of their searches , which depended 
apon the r curriculum topics to be covered. If a project focused on a topic 
su&h as career education or inservice training, where R&D information was 
less easily available/ the knowledge-base staff had to conduct a wide search 

for the needed products. Moreover/ such products did not necessarily exist 

* v • 1 

in large numbers* because little R&D work may have been done on the topic m 

x \ • 
the first place. Building a knowledge base, in these cases , became a chal- 

• 

lengmg task. - ' 

•_ • ^ ■ ■ , 

The projects- aLso vari^oTTri tfie~exten5i-veness of the ^earch-proceuur* 

-«v.?r time, One project (The NETWORK) assembled "its. entire knowledge base 

is part of its original proposal — i.e., before the initiation of the PSQJ- 
■ * " • 

«?ct~ and the composition of the knowledge base was modified only once after 
that. Other projects (Georgia and Michigan) required long start-up peri- 
ods and were, only completing their knowledge ba^es by the third year of the 
project, still other projects assembled an initial array of products, and 
th<en added to this array over titae as new products were recommended or en- 
countered (Pennsylvania)! Often/ these recoaaendatiort came from the sites , 
which, 'as'a result of their problem-solving activities {see Chapter IV of 
this report)/ had identified sotae product that was not part of the original 
knowledge base* * m 

Validation Procedure * Given the (array of sources searched, each 
p^6*3)fct 'as Expected to screen candidates for the knowledge ba£e on the 

basis of three criteria. First/ the product had to fafl within a relevant 

•* ■ 
curriculum topic. Second/ the product had to be in usable condition— -i.e. , 

the materials had t;o bo complete/ adequate instructions had to exist/ and 

both had to be available for purchase (potentially by several sites) from 

sooe source. Third/ and raost important to Ghis P*pe*/ the products had to 



represent products that had. previously been validated for their effectiveness 
and transportability through empirical testing* The ?irst two criteria were ( 
Jpm&led with little difficulty by each project . the last criterion, however, 
posed many difficulties and deserves, further discussion. 

According to the RFP, the validation criterion could *be met by tlje fact 

f 

that a product came from some acceptable! source* An acceptable source, as 

% » * \ * 

suggested^y the RFP, qould be an R&D lab or center, or art existing catalog* w 

The catalog 'of NlE-funded products is prominently mentioned injthe RFP* Other 



relevant catalogs were the catalog of NDN products or some substitute proposed 
,bj the project . In each case, it was presumed that appearance in one of these 
catalogs meant that a product had already passed 'through some validation pro- 
-cedure* This is true of -the NDJj catalog, for instance, where prp£ucts can only 
be listed,^ t*hey have beeh approved, on the basis of empirical evidence, by a 
USOE-NIE Joint Dissemination Review Panel. The panel, consisting of represen* 
tatives from USOE and NIE, judges products using \hree Conditions: 

[ * • The evidence of a product's effectiveness must be valid 

^ and reliable; ■ \ 

« The effectlsust be" of sufficient magnitude to have 
educational importance; and 

iy 

• The intervention and its effect must be reproducible 
at other sites (USOE and KlE*, 1977)^ 

Evidence of these three conditions is judged by six criteria v First/ 
a positive change must have occurred. Second, the effect -('or the, change) . 
must be consistent enough^and have been observed often enough that it is 
statistically significant. ^ Third, the effect must be educationally signifi- 
cant. Fourth, the intervention must bo. transportable. Fifth, the effects 
must a*result of the intervention. And sixth, the evidence of the product's 
effectiveness must be reliable. In contrast to the NDN procedure, however, 
the products listed in tiie NIK patalog, by NIE's own admission, had not all 
passed through a similar validation procedure.** Thus, it is likely that RDU 



/ 



* Of the 660 products described in the NIE catalog, approximately half 
"have undergone small-scale controlled tests for effectiveness." An 
additional one-third have been replicated in a number of sites "to ( 
confirm that the products are effective, can be used with little dif- 
ficulty, and work in a variety of settings," leaving approximately 
lit) products in the NIE catalog that may not ha#e been validated in 
* any way '(Catalog, 1975). In addition, an informal NIE review of the 
4juality of product evaluations suggested thit aany products classified 
as validated would not be able to pads a Joint Dissemination Review 
Panel review (personal communication from M. Millsap). 

15 



projects were already using nonvalidated products by using products from these 
presumably acceptable sources* 

As suggested in the RFP,, some projects established their own review or 
validation procedure (NRC) or used procedures that already existed in their 
states (Pennsylvania, The NETWORK, and Michigan}. These procedures varied, 
however, in their degree* of , adequacy as validating procedures. If thlf vali* 
dating partel, for instance, had access to empirical evidence concerhing the 
-effectiveness and transportability of a product, the review of such evidence 
constituted an acceptable procedure * In contrast, if the procedure only in- 
volved the gathering of an expert panel --which examined a product for its 
„ "surface yalidity* but had no access to actual empirical information- -the prp- 
cedure ^was not considered satisfactory. Based on cur interviews^ the adequacy 
of the validation procedures that were used by # these four rdu projects cannot 
h« judged precisely, all four probably had to contendvlth d^Fferm^ amounts 
of empirical evidence, depending upon the product under review* and we do. not 
* now t ncy with which only "surface validity" judgments were made. 

However/ a few .nonvalidated products probably entered the knowl^jjge bases be- 
A oau^e of the unevenness of thd vaWSation procedures. A 

Pioi^^ts could also enter the knowledge ba^e through other -channels, as 
happened in several of the projects (Florida, Michigan, Georgia, and NBA)* 
Thus, a product could emanate from some non-acceptable source/-! .e. , not a' 
lab or center and not one- of the previously mentioned catalogs — and also not 
be subjected to a state or local validation procedure. RDU projects were 
often forced to use th*s third procedure when an adequate array of validated 
products could not be assembled, and when a site would suggest sone product, 
it wanted to adopt. 

For most of the RDU projects, the knowledge base was assembled by using 
all three procedures, to varying degrees** The following paragraphs describe 
the general procedures developed by each project! 



* The RFP also mentions the possibility that validation procedures. can be 
applied after a site has adopted a product, and, that this would consti- 
tute an acceptable certification procedure. One project (NEA) vas still 
trying to ascertain thej^tatbs of its entire knowledge base, hovever, 
into the third year of The project. 
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Florida . The Florida ^Q^ect did not have its own ^validation proce- 
dure ^but selected ftjpodutts f orbits knowledge base from thd NIE catalog, 
the NDK catalog, and publishers, 1 listings of validated products. When 
no validated pjjpduc£ hatched a site's needs, the knowledge b$se staff ac- 
cented a nonvalidated product into, the knowledge base*. These products, 
although* they may have had a surface validity, had not been systematically 
evaluated for their ef feet ivenegg^ and transportability. * ^ 

Pennsylvania . The Pennsylvania knowledge base acquired products froo 
three sources — the NIE catalog, the NDN catalog, And the' listing of 
products in the Pennsylvania Diffusion. Program (POP). ' The knowledge base 
did pot have its own validation procedure, but relied on the validation 
procedures of the source* from which its* product* we r^ ^rawnr -She PDP 
validation procedure was less rigid in evaluating a product's effective- 
ness than the procedure used by the JDltP. The PDP panel/ cooposed of 27 
people from local school districts, state education departments, and in- 
stitutions of higher education, ^accepte^ a product as valid if indirect' 
evidence of the product^* effect wasprtiown^ 

NEC . The „NRC s knowledge base consisted of products from the NIE cata- 
log, the NDNWtalog, the Right-to-Read catalog, and E^C, *nd products that 
were acquiredMArough informal ;epurce$. Although many of these products, 
were validated by a stqte or n&tJional review pane^ the knowledge base staff 
also reviewed all candiSate products, using a consistent set of criteria. .* 
Each prtflUcflHde rwent two levels 0* Review. The knowledge-base staff first 
checked to make"sure that all the necessary prpduc/ information was avail- * 
able. Zf it was,, a six-member panel, comprised of the NRC pro'ject director, 
tt^e >nowledge-base# manager*, the Right-to-Read director for the state in 
which the panel convened, a teacher, a representative from higher 'education, 
and the linker from the host state, then rated the product on its proven ef- 
fectiveness* Those ^>ro4ucts receiving a minimum score or better were accepted 
into the knowledge base. The Nifc pro ject* adhered to this procedure throughout 
the project and appears to. have had only validated products in" its knowledge 
base * 



• — « • m 

Michigan . The Michigan pro'ject acquired its* products from th£ HIE 
catalog/ the NDN ca1£*og, commercial publishers, and through worjJ-of -mouth 
communications. Ori£iAally, «ft^Products that were accented' intb. the knowl- 
edge base had'to satisfy thrfce criteria. Firsts each product had to be re- 
viewed' by the^JDRP. Second, the product's effectiveness had to have been 
tested at an Appropriate site* Third, 1 the product had to have been approved 
by thk State -Classification Committee. .The^e criteria were relaxed, however, 
when few validated products in career education were found. According to one 
interviewee, only 20 t percent /jtff Michigan's knowledge base products ultimately 
satisfied the project's origjnuil validation criteria. 

* 1 . *U ' * • . • , 



* The Florida knowledge base acquired* the SRA kits— Mat foema tics Involvement, 
Mastery Test/ And Schoolhouse Comprehension— under these, conditions.' As 
of the spring of 1979, three sites in Florida ha<J adopted these products. 
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^ Georgia . ' The Georgia project did not have a/validation procedure. Al- 
though the knowledge base staff reviewed the candidate products for surface 
validity, making sure "thAt the products' materiaL^Pfcisted, that the products 
were easy to use, and that they came with staff development assistance , 
proven success and transportability of the products were not assessed. Some 
of the products were selected from the HIE catalog and the KDtf catalog and, 
thus, may have been shown effective by other review panel*. Y^t the Georgia 
project it Ail f hard no systematic procedure for judging .the prodqet's validity, 
and raanyvoroducts w6re acgu\yd through sources other than these catalogs, 
such as listings of Title IV-C adoptions in the state. 

The NETWORK . The Network selected its products from the NIE catalog., 
the NDN catalog, and ERIC. A few products Vere received through education- 
al labs or centers* All products were validated* Those products from the 
.NDN catalog were ^accepted^a^ validated^ .JHi_.others had been reviewed by 
state level panels, where statistical evidence that a product had a posi- — 
tive effect on student^, teachers, or administrators had been 'presented. 

t 

,NEA . The NEA project did not have a procedure for assuring that only 
yjtlidaSfed products weVe admitted to the knowledge base . Although, some 
products had been field tested as part of their development, most of the 
615 products^ in the knowledge base had not been validated. As with the 
Michigan project, many of, the products were acquired through comer cial 
publishers and through recommendations of personnel in education associa- 
tions, and were not systematically checked for their validity.*" 

Results of Knowledge Base Acguisitioh * 

The results of these acquisition procedures are reflected by the 
proportion of validated products in the seven knowledge bases. To the 
extent that the NIE or NDtf catalogs were N the source of * the products, the * 

* 

proportion of -nonvalidated products could be checked (about 100, of the 

NIE catalog's products had no validation data--see footnote p. 15), 

and these NXfe^KDN results are shown in Table 4. However, although product 

lists were created by all projects, we have been unable to make a defini^ 
* 

tiVe analysis of the remainder of the products for two reasons . First, v 
the projects themselves did not sufficiently document their own valida- 
tion procedures for each product* For any .given prodSct that had passed 
through the local. validation procedure, for instance, we^do not know the * 
full nature of the evidence that was reviewed. Second, the knowledge 
bases in the aggregate had, too many different products, so tjyvt any retro- 
spective attempt to call de^lopers at this timeTwould be impractical. 



* In the spring of 1979, NEA'-s knowledge-base staff was palling developers 
to check on products' validity, even though most sites had already se- 
lected and adopted a product. The" results of the phone calls showed 
that many products had not been empirically tested. In gome cages the 
developer had moved from the institution where the product had been de- 
veloped, ,and the developer's new address was unknown. In addition, 
several of ^he publishers were no longer in laziness. 
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Table 4 

Number of Products Proa Either NDN or NIE Catalogs , 
by Knowledge Base* 



Knowledge 
Base 



H\ Total Nq* 
.of Products 



Source of Products 



Non-NIE or 
NDN Source 



No. 



# NIE 
Catalog 



No. ' 



NDN 
Catalog 



No- 



No. of non- 
validated 
NIE products 



Florida 
Pennsylvania 
NRC . 
Michigan 
Georgia • 
The NETWORK 
NBA j> 



) 



92 
123 
59 
150 
115 
41 
.615 



50 


54.3 






17 


13.8 


20 


33.9 


120 


80.0 


46 


• 40. tf 


17 

• 


41.5 


? 


? 



r 



27 » 


29.3 


15 


16.3 


"(5). 












38 


30.1 


68 

*> 


S5.3* 


(7) 


6 


' 10.2 


33 


55-9 


(0) 


24 


16.0 


6 - 


4.0 


(24) 


21 


18.3 


48 


41.7 


(2) 


" 6 


14.6 


18 


43.9 


(0) 


? 


? 


? 


? 


(?) 













the figures in this table were compiled by determining whether the products in each 
project f s knowledge ba$e, as of spring 1979, were listed in either the NDN or NIE 
catalogs. 
\* 
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Nevertheless, we can make rough estimates of the -degree to which the 
* v • 

knowledge ba£es contained npnvalxdated products, this can be done by sum- 
narizintf th« acquisition procedures/ as shown in Figure 2. Each chart 
depicts the pp>ce$iures fc^lowed by each project, dividing fehe' procedures^ 
into thpee cafregori^^ products entering as a result of an acceptable * 
catalog (fcolunn 1); products passing through a state or project valida- 
tion procedure (column 2); and products cooing fr6m outside sources 
Icoluiwi A, vertical arrow xs shown where products entered the knowl- 

edge base* through one of 'these three procedures /. The products from out- 
si4e sources .should prefcfrably have passed through the state or project 
validation procedure, and where this occurred (NRC, Pennsylvania, The 
NETWORK, and Michigan) , a horizontal arrow frs, shown .* ' 

iThe charts show that An three projects (NRC, Pennsylvania, and The NET- 
* WORK/ \he acquisition procedures were sufficient to assure adequate screen- 
ing, given two .assumptions % 1) that the Nip products had been among those 
validated by NIE, and 2) 'that the local^validation procedures were properly 
followed for every product. iv In Pennsylvania, however, 7 of the 38 NIE 
products were those that were nonvalidated. ) Two projects (Florida and 
Georgia) had no, validation procedure of their own. yet products from outside 
sources were known to have b*en used*** This seems to have occurred to a^ 

ouch greater extent in Georgia than in Plorida, because Georgia had to cover 

**■ » 

topics, suph as career education and district-level planning, in which few 
R&b products were known to have existed. Finally, two projects '(Michigan 
and- NEA) are known to have acquired many products that neither came from an 
acceptable source (column 1) nor passed through a validation procedure (col- 
umn 2), even though ^ich^gan did have such a procedure. This is reflected 
by % t^$ double arrows in column 3. , 



u r . v . -. n 

* # It may (be that products from outside sources have nevertheless passed 
* through some other validation procedure* Although the ertteht to which 
this ocburred rs not Jchown, ve do not believe it ^occurred with signifi- 
*cant frequency to alter our overall conclusions. 

♦'Florida, although >t had no formal validation procedure, did screen 
-.products in a more internal sen6e. For example, products that were 

authored by\cxperts or conformed to current theory were admitted to j 

the knowledge base. 

' ' 30 . 



Figure 2 
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These charts suggest that the seven knowledge bases had the propor- 
tions of nonvalidated products shown in Table 5* 'Based on these figures, 
the t RDU program as a whole appear? to have allowed the use of many such 
products. In fact, the definitions of what constituted an "acceptable" 

product shifted over the course of the project as the seven projects and 

* * * 

the HIE program manager confronted soqe of the difficulties and perceived 
the real constraints involved in developing a ^ comprehensive array of 
products to serve school needs* At, the inception of the program, the 
emphasis had been placed upon BSD products , or upon new curricula* or 

> other innovations that were part of a deliberate applied research, de- 
velopment, packaging, and dissemination process. In the RFP, as ire have 
noted, ,£he definition was shifted somewhat to include products that were 
Empirically validated , whether A&ey were based upon research knowledge or 
not* Finally, during the course of the project, the HIE program of f ice r 
and the seven project directors came to a decision that the development 
of , less rigorous screening procedures would be acfceptabip (although if 
the project concurrently instituted procedures to empirically evaluate 
the effects of the Innovation the less rigorous would not be preferred). 
This decision was reached af te* nuch debate azaong the project directors, 
and Specifically after the Florida project asked whether\it should oeet* 
client, demands for diagnostic-prescriptive innovations in reading despite 

^ the questionable quality of these products using "validation" standards 
(letter from R. Gagne to T. Israel, October 11, 1977) t This decision 
was not, however, made until after the project had been in operation for 
nearly a year and a half, and after nuch effort had been expended on an » 
attempt to' meet more rigorous stancUrds of quality control- * r 

Before drawing any final conclusions about quality control, ve must 
examine the projects* procedures for matching site needs with anilable 
products* One possibility, for instancy, is that the sites ended up 
adopting only the validated products in the knowledge bases, thereby com- 
pensating for the knowledge base slippage. Bowever, this turns out not 

„ to have been thcr case. 



22 



t 



ERJC \ 



32 



/ , * Table 5 

Proportion of Nonvalidated Products in Seven Knowledge Bases 



f Nonvalidated 
Products in 
Knowledge Base 




• 

Project 


» 

None 




NRC, The NETWORK 


• 

Soae 
• 




Florida, Pennsylvania * 


Kany 




Georgia 


k majority of all 
f products in the 
\ knowledge base 

/■ * * - " 


« 


Michigan , NEA 

• 

* 

• 
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s > CHAPTER III 

MATCHING SITE NEEDS WITH AVAILABLE; PRODUCTS 



The ^a-xOf products in the RDU program depended on two major steps* 



lAfi-xOf" pr 

The first, described in the previous, chapter, was the development of a knovL- 

r * B 

edge base* The second, described hire, was { the provision of assistance to 
school sites—to .guide these sites in matching needs to specific products 
and then to assist in adoption (and later implementation) • Much of . this 
assistance was initiated while the RDU projects were still developing their 

/ 4 

knowledge bases , so that these two major steps often occurred simultaneously. 

The. work with the sites was a cooplex process that could involve several 
activities, such ass 

• identifying, candidate sites to participate in the 
RDU project (in scoe cases, sites were named in the 
original proposal); 

m establishing ad*hoc teams or cotoaittees at 'the site 
* y level and, where relevant, providing sites* with funds 

, to conduct the FDU activities (funds wer^ used to 
* * purchase released tine for teachers or # to purchase 
the products themselves) ; 

« recruiting and training individuals to serve as 

linking agents , usually located in some interaediary 
organization, to assist sites in selecting and imple- 
menting ^products from the knowledge base; 

- * requiring the sites to follow a problem-solving 
process , which included the examination of needs 
/ before considering alternative courses of action; 

and 



m assisting sites in making the final selection of a 4 
specific product to bo used* 

Because of the complexity and importance of each of these activities , sep- 
arate reports # will be written abo^it several of them as part of the overall ^ 
Abt Associates research effort* Moreover, the success of the RDU program 
cannot be judged on the basis of any of these activities alone, as the' de- 
sign of the ,RDU program had multiple objectives '(see Corwin, forthcoming 
J.980). The main purpose of the following discussion is to continue our 
tracking of the selection and use k of educational products, with the primary 4 

focus oi> the last activity listed above* . + 

< 

\ ■ 
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The selection proces? involved several steps, culminating with a 
specific adoption; these steps as<well the adoption out cooes are 
described in the following subsections. *" CXjj attention will again be 
on the' degree to which* nonvftlidated products were ultimately adopted, 
and thus the extent to which the adoption patterns did not conform to 
the original intentions expressed in the RFP. The steps in the selec- 
tion process included; 1) the initial presentation of candidate pro£-> * 
ucts for adoption! 2) the screening of these candidate products by the" 

sites f and 3\ the final adoption. * 

t - 
Initial Presentation of Candidate Products for Adoption 

1 K * 

Assuming that a site had completed »soae type of needs assessment, \ 

it was then ready to, consider various products* that could bcuadopted. * > 

The original RPP, however, provided no guidance for how the matching ^ 
process--in which site needs were to be related to available products-* 
was to take place. Moreover, this matching process was a potentially 
difficult one, if ther needs of a site could not be matched by any Avail- 
able product. Most of the projects ultimately developed a general pro- 1 
cedure wnereby lists of potentially relevant products from the knowledge 
base were made available to the site teams for their consideration. How- ^ 
ever, there were significant variations among the projects irf following 
this procedure. ■ 0 \^ 

, Modes of Corounication . Inmost cases (Florida, Georgia, Michigan, 
and NE£), the site teams were encouraged to make direct contact with 
-Aknow ledge-base staff, 1 ** indicating the topic or topics of interest that* 
^ resulted frca their needs assessments * The knowledgfe-base staff then 

identified an appropriate array of potentially relevant products and sent 
brie'f descriptions of these products to the sites. In the NBA project, 
this w^as done when a site would make a telephone call to the knowledge- 
* .base staff and descriptions of all the products in the -relevant categor- 
ies, based on a prior classification scheme, were then sent to the. site. 



filcati 



* Modifications could also occur after adoption, but thdse implementation 
activities are beyond the scope of the current report. 

+*In Michigan, site? contacted the knowledge-base staff in the state de- 
partment of education. . . ^ 
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*In other cases, and dttring the Initial phases of most projects, the pro- 
cedure %#4s moje interactive and involved face-to-face communications, with 
the site staff visiting the knowledge-base 8t$ff or vice versa. a In Georgia, 
some initial orientatlon^ta the whole product array in the knowledge, base 
was communicated through two " educational exchanges, 19 or conferences at 
which teachers frota mny sites were invited .to review a wide array of date- 
rials. The£e_ conferences were uniformly judged to be a F highly satisfactory 
way of orienting site personnel and giving then an idea of the potential 
.products before the needs assessments were completed. 

In other cases '(The NETWORK, JTOC, and Pennsylvania), the site teams 
did not make direct contact with the knowledge-base staff, but worked in- 
stead with the project's linking agents In these cases, trie linker 
ascertained the topics of interest from the site, worked with the knowledge 
base staff to select potential candidates, and then explained the vajuous 

possibilities to the site personnel. The linking agent thus served as an 

v 

intermediary i^^e matching process. - , 

This aajoi variation in' commtfaication links did not appear to create 
any consistent differences in the Salt irate product adoption patterns, but 
deserves further attention because of the different roles implied for the 
lin^in? agent. In the first mode. of communication, where sites dealt di- 
rect 1/ with the knowledge-base staff , the site personnel were Regarded as 
the primary users of the knowledge k^ase, and knowledge-base documents were, 
oriented toward the terminology aAd needs qf practitioners. The linking 
agent played only a secondary role in the communications process, generally 
being informed of the site's interaction with the knowledge base after it 
had occurred. In at least one project (NEA), linking agents cane to play 
increasingly peripheral functions^ as a result of this* procedure, and on oc- 
casion the linking agents were no% even informed about the site's cccuauni- 
cation with the knowledge base". In the second n&de of communication, where 
sites dealt through the linking agents and only indirectly with the knowl- 
edge-base staff, the linking agents were regarded as the primary users of- 
the knowledge base. In this 6itu£t±on, knowledge-base documents were oriented 



# 



* A "mixed" mode of communicatiofis could also occur on occasion {e.g., 
in a few Sites in Georgia and HRC) , in which* the knowledge base st;aff 
and .linking agents worked together in dealing with site personnel. 
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toward the terminology an3 needs of the linjjing agents, and thus the 
linking agents had a strong role in influencing- a site's final selection. 
Overall, the node of communication reflected the degree of activity pn 
the part of the linking agents; vhpr^ they were not heavily involved,' the 
knowledge-base staff ^Actually filled the linking function, this was the 

,case for several of the^projects. m / * 

Size of the Candidate Array. Each project also had to develop" its 
own sense of the appropriate size of -the initial Candidate atfray, whether 
presented directly to the site or through a lining agent. The size, or 
number of products, had t^be large enough so that sites could have some 
choice in making their ^fejjtl selection, but small enough to be manageable. 
Projects generally pneseSted about ten products, where available, in this 

^initial array. This problem was further compounded by the nature of the* 
materials used to describe each product. Most sites would have preferred 

receiving the product itself, but would then have had difficulty in review- 

1 

ing the materials for such a large array. As a result, all the projects 

* ■ ■» ■ 

developed their own one- or two-page descriptive summaries of each project, 

and this was the material that was sent to each site.* • ■ 

■ ■ * 

Screening of Candidate Products 

Sites* screening of candidate products generally followed two stages. 
During the first stage, a site, in theory, was to review the candidate 
products for their potential relevance, and to reduce the initial pool of 
candidates to a smaller set of two or three final candidates. During the 
second stage, the site would request more information about these final, 
candidates, and the knowledge- bas^ staff Would have to fee contacted again 
for this information. At this point, the knowledge base staff typically 
loaned the actual product materials to the site, or even suggested direct 
contact, between *the site and the original developers of the products (e.g., 
Pennsylvania). The site teams "then reviewed in detail these final candi- 1 
dates and selected one for adoption. 



* The issue of the size and nature of materials in the initial array needs 
to be given, greater attention in the future. Pennsylvania, for example, 
initially presented a large number of candidate products to sites* When 
.sites were finding it difficult to .select a product free* sudh a large 

— atrayv-fcho knowledge ba se st a f ^ significantly reduced the number of 
candidate products that they presented. 
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this secortd stage was conducted cost systematically in two of the RDU 
projects (Georgia and Pennsylvania) . Occasionally however/ the knowledge- 
beae^ staff had difficulty keeping track of the product materials, that had 
been loaned out. In the other projects, the second stage was of ten blurred 
with the first, so that the process of narrowing down the initial list of 
candidates occurred in aS^ss distinctive, two-stage manner. For instance, 
where linking agents were heavily involved in the matching process, the two 
stages might have been collapsed into a single stage, or several iterations 
might even have occurred before a final adoption was made. 

Whatever the process, the final selection often revealed a host of 
problems that had to be addressed in a j£ite-by-site manner. First, in 
some cases the sites may have had a single product in mind all aloijg, pos- 
sibly preceding the needs assessment activity. Ihis prior preference may 
have been well-suited to the site's ow sense of its needs before entering 
the RDU program, or it may have reflected a bias that was not justified by 
the site's ^actual needs. The extent to which these situations arose can 
only be documented by further information from the sites.* 

Second, in other cases the sites may have identified their own candi- 
date products to augment the candidates selected by the knowledge-base 
staff. This was especially true in those projects (e.g./ Michigan, Georgia, 
and NEA) where it took two or three years to complete the knowledge base, , 
and where the sites had therefore advanced more quickly than the RDU pro}- 
ects' preparations, in these cases, the sites' candidates were frequently 
incorporated Into the incomplete knowledge b&se* These cases represented 
important occasions when nonvalidated products could be incorporated into 
the system, because the sites did not necessaxj.ly present any ccapelling 
evidence that their candidates had been validated for prior effectiveness. 

Third, the review of candidate products often revealed a mismatch be- 
tween the categories or terminology used by the knowledge btjses to classify 
their products and the categories or terminology used by the sites. The 
level of specificity could be different. Thus, sites cpuld decide in their 



* At the time of this report, survey data frocv the sites were not yet 
available for analyzing this issue. 
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needs assessments that they had a "mptivatioM** problem among the students/ 
which was not specific enough to identify acc^irately^the potentially rele- 
vant products (e.g., Florida); further probes were needed to .determine that 
the specific problem may have been students fighting in the hallways, and » 
on this basis it was easier to determine whether x a relevant product was 
available or not. Conversely, some sites came up with _sgecif ic : needs state-, 
ments, but then could not easily cope with the generality of the product 
descriptions (e.g., Pennsylvania). Similarly, some sites made their needs 
known in terms of curriculum content even though the knbwledge-base products 
were initially classified by teaching processes (e.g., NEA), or vice versa 
(e.g. , Plorida) . 

Fourth, it was entirely possible that^one of the candidate product s7 
even with accurate communications, served a site's needs. In theory, the 
knowledge-base staff was then supposed to conduct a further search, beyond 
the knowledge base, for a potentially relevant product—a provision that 
was covered by the original RFP (HIE, 1975:15). However, little attention 
had been given to the fact that this wider search could take^ a long tune- 
far exceeding ttfe site's schedule for adopting a specific product— and this 
broader search would have to be conducted with undue haste, again leading 
to the possibility of using nonvalidated products. Whatever the outcome, 
the provision for having a further iteration a* this point was judged un- 
realp.stic-by some projects. 

All of these problems should suggest that ,the screening process was 
not an orderly or simple one. In fact, we believe that this process bore 
the brunt of one of the conflicting elements in the basic design of the * 
RDU program.* On the one hand, sites were to use existing R&D, products • 
On |^ie other hand, sites were to undergo a problem-solving process, where- 
by a needs assessment was the^rftial step.** Only unabashed optimism 
would lead to the conclusion that the available validated products were 
likely to match, with high frequency, the articulated needs of sites. As 
a partial remedy, the^TP did make one provision for dealing with the po- 
tential conflict x 



* There is evidence from related interviews that some HIE staff members 
were aware of this potential conflict but could not influence the modi- 
fication of the design of the itou program. 

**The conflict between these elements may be found in programs other than 
education (e.g., see Vin, 1978; and Roessner, 1979). 
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. .. a, legitimate, project outcome could be the conclu^ * 
,*ibn tjhat in a specific local situation there is no 
RfiD {product] that represents an acceptable solution + 
to ,the defined problem. Such a conclusion, properly 
documented, could add to (KIE's] understanding of 
field requirements for further R&D (KIB> 1975:4-5). 

This provision h^ils, however, to indicate what the sit* should have done when 
this situation was encountered. Most, if not ail, of the sites were recruited 
into the RDU program on the basis that *&ooe assistance would be provided in **** 

m % •* 

dealing with their school problems. An impasse of the sort described in the 
RPP, whiJLe potentially useful to NIE, would not likely have been an acceptable 
concjusibii from the site's point of view. Not surprisingly, the seven RDU ■ 
projects therefore did_ everything they could to fi^nd soae^accep table product 
for every site, even when the impasfce was encountered. At tf\e same tinted it 
is also true tha^the projects did Uttle to document, on 4 systeaatic basis, 
those situations in wh>fch the impasse occurred, and thus there is only sparse 
information regarding further needs fofnew types of products. The only in- 
formation of this sort derives from ..interviews with project director orTcnowl- 
edge-ba?e staffs, who typically reported ,the need for more products in second- 
ary .education in general, and in non-reading and non-math curriculum topics. 

Final Adoption Patterns * 

As, a result of this matching process, tjte sites finally did adopt some 
product..* For sites that had adopted a product by the spring of 1979,** the 
full list of adopted products, by project, is shown in Table 6. The table 

shows the produrtTs by educational area (career education, reading, mathema- 

* / 

tics, and miscellaneous). Sixty-four products were adopted by only a single * 
• «• • 

site, whereas 36 products were adopted by. more than one site; of these multi- 
ple adoptions, the mos% popular products were; Wisconsin .Design for Reading 
(adopted bfr^pleven sites), Career Development Centered curriculum Cseven 
sites ),/att Diego R2R (seven sites), Exemplary Center for Reading instruc- 
tlo^ECRI (six sites), It Works (six sites), AEI* Career Decision Making Pro- 
gram (six sites), and Houghton-Mifflin Basal Management System (five sites). 

• c ' 

* Some sites dropped out of /the program before reaching the adoption stage; 
others dropped out after adoption, but little documentation exists regard- 
ing the reasons for these dropouts. (The Michigan project did collect in- 
formation on the reasons for sites dropping out, however.) Finally, some 
sites had not reached the adoption stage by the time this report was drafted. 

**Many sites had not reached the adoption step by t}iis tiiao. 
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^ * Table .6 

Product Adoptions for Each RDlf Project#*by Spring 1979 




NAME OP PRODUCT 



V 



CAREgR LOCATION 

Crisp Co\ <Ga») 6 Olrange Co. 

, (Fla.) Career Education* 

■ * ■ 

Basic Skills Through Practical Arts 

.> 

Georgia Career Guidance 
It Works , 



JS Locally Developed Career 
^ucatipn HandbooK* 1 



Orange Co. (Fl/u) \pareer Education 
, Free To Be. You and Me' 



"7 



Careens a Supplemefital 
Reading program 

'innerchanga, • 

Project -HEAR (1L 
* * m 

• Livelihoods 

My Bread and ButteMUes 
i Career Book 
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Florida 



PROJECT 



Penn. 



NRC 



Mich. 



Georgia 



2 
3 

2 
1 

5 
3 



U 

1 
4 

- 6 

f 



NETWORK 



KEA 



v. 
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HAME OF PRODUCT 



CAREER EDUCATION (continued) 

Career Awareness Exploration 
Curriculum Kit y 

Preostyle 

Career Developnent Centered 
Curriculum (1) 

Getting It Together 

**rhe Job Ahead 

The* Magic Circle- 

Ewployability Skills 

AEL Experlencfc-Based Career 
Education Program (1) 

AEL Career Decisionmaking 
Program 

_ First Jobs * 
■>» 

4 

M 

Healths Decisions for Growth 

Career Exploration in the 
Earth Sciences (2) 

Michigan ACT Career Planning 
Pro gran 
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Table 6 (continued) 



PROJECT 



Florida 



—3 

Penn, 



mc 



Mich. 



1 
1 

2 - 

3 

1 



3 k 

1 

1 

1 



Ge'orgia 



NETWORK 



NEA 



t 9 
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PROJECT 

• 




• 






i 




HAMS OP P FDD OCT 

• 


Florida 


m 

Perm. 


NRC 


Mich. 


Georgia 


NETWORK 


KEA 


< 




CAREER EDUCATION (continuted) 








* 


• 










Working With Trees (2) 

• 


• 










■ 


• 


• 




> Introduction to Coamurrity 
Centers 




■ 

• 




i 












Career Opportunities Boxes v 

• 




• 




2 












Just Me (2) 








1 








n 




Career 'Development for 
Children 








I ' 

1 




• 








Exploring* Careers 




f 


• 




* 


• 

( 








" Valuing Approach to Career * 
Education 








2 


• 


\ 

I 


• 


• 




Goofy Goes to Work 










> 


• 




V 


Kangaroo Kit 


■ 




• 


• 
1 

1 


• 


p 






— . 

t 


^Sqso Kits i^JA^rici^^^^^ 
''TiTT'tim^n ftffnnr,)* ^^^^^^ 

Me and Others \ 

^ J 


• 






4 2 

2 


: I 

* 


• 

• 

• • 
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Table 6 (continued) 



.NAME OP PRODUCT 



PROJECT 



Florida 



Penn. 



NRC 



Mich. 

r 



Georgia 



KETWOftK 



READING 



4 



Continuous 'prbgress 
Reading * 

English/Reading Rotation 
Program ( 1 ) 

Project pal (1) 

Secondary Reading Lab 

Individual Language Arte (1) 

Manilas prograa, NY State 
Coordinated English- " * 
Language Art a * ( 

SMART {1) 



Pr*ci*i©n T»*cfain<3 «ft4 SOR> 
Perception Plus (1)' 

fiDR (1) ' 

Basic Spelling 

Wisconsin Design fob 
Wding <2) 

Project KARC 

Open Court 



Exemplary Center for 
, ' Reading Instruction— 

erJc* 5 " » 



1 

3 

2 
4 
1 



1 

2 
2 

1 

2 



■ ') 



1 - 



1 



- 4 



♦Table 6 (continued) t 



'a 



NAME OF PRODUCT 



PROJECT 



Florida 



Perm. 



v NRC Mich. 



Georgia 



NETWORK 



NEA 



READING (continued) 
' tome 11 Loft Skills 
^SRA^Sqhoolhouae Comprehension 

PCCUS—Reading- Curriculum* (1) 
, La u bach Tutoring Prog^ia* 

Junior Great Booki* 

Content Reading Insetvice 
Package* 

Coonunication Skills Improvement 
w Center (1) ^ 

Intensive Reading Improvement 
Center iJL) ^ — * 

Dallas Basic Skills* p 

.Nanpa Communication Skills* 

r 

Miscellaneous Products 
developed by the sites* 

San Diego R2JT^ 1 

Pegasus-PACE (1) 

Classroom Intervention 
Project (1) 
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Table 6 (continued) 



NAME OP PRODUCT 



PROJECT 



Florida 



Penn, 



NRC I 



Mich* 



Georgia ■ 



NETWORK 



NEA 



READING *( continued ) 

Systematic Approach to 
Reading Improvement^- SARI 

Houghton-Mifflin Basal 
Management System* 

Spel 1 scrip t* 

Parents Assist Beginning 
Reading Management 
System* 

SWRL-Ginn Communication* 
Skills Program 



Sustained Silent Reading* 
Reading Resource Center* 



Development of a Language 

Arts Scope/Sequence* 

•* 

Engleman-Bockor Reading 
and Horphographic 
Spelling* 

Columbia River Developmental 
Reading Program # 



Bay Area Writing Program* 

Holt Series and Management 
System* 

ERIC ' 51 
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Table # (continued) ' % 

V 



s 






PROJECT 










• 

• 

" NAME Or PRODUCT 




Florida 


Penn . 


\ 

NKC 


men * 


Georgia 




MTU 


READING (continued) 

Hawaii English Program (1) > 


# 


• 






• 








Individually Programed • 
Instruction^) 

• 


















Project Catch-Up (1) 










* 


* 




* 


Accountability in Primary 
Reading Program 
















- 


Vocational Reading Power 
Program { 1) 


• 










- » 


V 

* 




AndoVer Individualized 
w Reading System 

-J 




- 




- 


* 


- 

■ 






Concepts £nd Language (2) 


















Improvement of Basic Reading 
Skills (1) 




< 






* 




1 




1976 Lippincott Reading 
Text and Management System 






■ 




• 




r 




MATHEMATICS 


















STAMM,{1) 






1 




t 


3 




* 


PRIMES 

» » 




» 


1 




m 


■ j 






Keymath 


< 








' * 








- Proficiency Verification 

Program With Learning f 
O Center Resources * I 

ERIC 


• 






• 






■5< 





Table 6 (conclude^ 



CSV 




> NAMF Or PRODUCT , 



MATHEMATICS ( coptjtnued)/ . 

- - ■ a * 

Bfevard County LAMP * 

Math Learning Systems 

^SRA—Mattheaatics^ InvolVement 

SRA-^astery Test 

LLANEOUS * - 




Evaluation Workshop frora*the-^ 



fit? Center for t;he 
* oS p Evaluation** (2 



Stu<?y of ' 



V 



SPECS; — School Planning^ , 

Evaluations fc toouaxlnicationa' 
,"syaten*\(2) , 




asitive Attitude Towards - 
Learning — PATL (1) 

Individually Guided Education (2) 



The NBA project did not keep a list of the products that wore adopted by the fUtea« 
* Product not in knowledge base 

(1) Product listed in the NDN catalog 

(2) Product listed # in the NIE catalog 
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Witi£jf^gard"to the the use of validated products ^it s t&* again impossible 

to determine the precise proportion of adopted otoducts tfelt were validated, 

yo dat^w^re .available to determine whether ,eaijh adopted product had been 

validated, -and the list is too long to be investigated on a post hoc basis. 

However , some summary inferences may be made b^following either of two al- m 

ternative approaches- 

The first approach Is based on the proportion of adopted products that 

■ 

fell outside of the knowledge base. la these cases, sites completed their 
needs assessment and identified some product th a t was not in the project's 
original knowledge base. Although .some,, projects tried to resist this type 
c>f adoption, either by dropping the site from the project or by disallowing 
the ssrali amount of funds available for assistance in implementation, 21.1 
percen^ of the adoptions ultimately involved a product that was *outsi,de of 
the knowledge base, while 78.9 percent involved a product that was inside 
the, knowledge base (see Table 7). Moreover, only about half of the adop- 
tion *nvoWir-g products within the project's knowledge base were in proj- 
ects *n*RC, The NETWORK, Pennsylvania, an£ Florida) whose knowledge bases 
were *iged to have had a or Itu ^ 1 proportion of nonvalidated products 
in nhe first place (see Table 5 presented earlier). Thus, if one assumes * 
^na* a*! of the products frcqt outside the knowledge bases were nonValidated, 
s-r.is ^fproach suggests that, overall, SO to 60 percent of all the adoptions 
may frage involyed^nonvalidated product* (21.1 percent plus half of 78.9 per- 
cent , . • ** ^ * 

The second approach is based on identifying thtf adopted products ac- 
cording to their, source tNXE catalog, NDN catalog, or neither), and combin- 
ang these frequenci-es with two known factors: the number of KIB products 
that had not been validated,* and whether the project had its own valida- 
tion procedure. These data are summarized in Table 8, which assumes that 
products subjected to a loctfl validation procedure may t*e considered vali- 
dated except for Michigan, where it was known that most of the products 
were not subject to the state classification procedure), and that produces 

t falling outside of the knowledge* base or not subjected to local validation 

t 

* There was some question during th£ program wh^er a product was accept^bli 
tt it was a researchTb^sed £rodu?t but had%not been validated. Folloying 
Uscussions with NIE, some projects *did allow this more lenient definition 

• and accepted products from the NIE* catalog into the knowledge base on this 
basis. f 



Table 7 

"Ptfpduct Adoption Patterns^ 
by Spring 1979 (sifc RDU projects*) 



KNOWLEDGE 

! . BAsr 4 



STATUS 




'NO. OF*' 
ADOPTED 
, PRODUCTS 



ADOPTIONS 



NO. 



j Outside o£ Knowledge"^ 




74 



.26 



100 



41 



194 < 



7B.9 



21-1 



100.0 



* The t4ble tops not present adoption patterns for the NEA, because'* 
a lx$t of .Adopted products was not maintained for this prp^ect. 

** Some srtfcs made more than one adoption. lyi 'these cases % each r 
adoptioh was counted separately. \ 

. / " * . 

***Thx* does not include 43 adoptions for which no information. was 
m available regarding the product adopted. 
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Table 8 



Product Adoption .patterns foe Each RDU Project , by "Spring ^^79 











ADOPTIONS 










I 

Total 
- 


Assumed To l 

\ , 


Jave^Been ^lidated 


Not Known To Have "Been Validated 

• 




Project 

' i V 


J 

Number 

from 

NIB 
Catalog 
(Validated) 


Number 
from 

Catalog 


* Number 
# Thr6lrgh 
Local 
Validation* 


Number 
from NIB 
Catalog * 
(Not 
Validated) 


Number 
* Outside 
of 

Knowledge 
Base** 


Number 

• 

Without^ 
Local 
Validation** 


Percent 

Hon- 
■"validated 


Florida 


22 


4 


i ' • 

• 


* 


• * 






• 7fr3" 


Pennsylvania 


*■ 13 






3 




l 




7.7 


KRC 


45- 




5 


6 




.30 




66.7 


Michigan 

• 


63 




9 




3 


3 


48 


„ 85.7 


Georgia 


24 


-1 

• * 


5 






4 




75. 0 S 


Network* 


27 


6' 


18 


"3. 








o.o , * 


NEA 


? 


? 


? 




? 


? 


,? 


? 

i 


Total 


194- 


> 

, 15 


47 


12 • 


4 


38*** 


4 

78 


61.9 



•Bach of the projects with a local validation procedure was arbitrarily assumed to hive properly used 
it, leading to a core conservative eatiisate of the overall proportion of nonvalldateU .products. The 
single exception is Michigan, where it was known that nx>st products did not go .through the local vali- 
dation procedure* > ♦ . 
•••Soma, of these may have been validated through an alternative proceduce le.g., by the commeccial publisher). 
The extent of this phenomenon Js not known. ^ . 
***Thifee products that were outside, the Knowledge base were nevertheless NIE or NDN products. These thcee 
products were, there fore J assigned to the NIB or HON columns in th'is table. 
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procedure nay be considered nonvalidated. Th£s aissumpti^ holds even though 
a few products may have been validated through some other , undocumented have 
been validated througtTsome other, undocumented procedure! . Based on this 
assumptx:>n, this approach suggests Jhat up to 60_percent of the adoptions 
could have involved nonvalidated products , 

Finally, a secondary analysis of data available free schools that had 
been visited* by either KDU project or Abt Associates research staff mem- 
bers indicates that only 64 percent were known to have adopted a product 
that nad been systematically field tested. The field testing did not have 
to neet any en^rical standards, and was simply an attempt to install the 
produce in a site or sites and determine whether it was useable. 

These different approaches yield estimates that indicate that by the 
sp?ic^ :>£ 1379, cany ofcthe adoptions lnvo^veS nco^aL^i dated and^even non- 
?ie*d-tvsted products. Although these are necessarily imprecise figures, 
the ie* alts are based on conservative assumptions. For instance, the final 
estimates exclude the NEA project, where it is known that the great bulk of 
'-he i3opt-.cns involved nonvalidated products. 
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CHAPTER IV 

DELIVERING EDUCATIONAL PRODUCTS T£ SCHOOLS ; CONCLUSIONS AND FUTURE PROSPECTS 

The preceding chapters have described the processes by which validated 

products were eventually adopted (or not adopted) by the school sites within 

' the PDU program. Ve have attempted to show that a good portion of nonvali- * 

dated products eventually made their way into the knowledge bases of several 

of the RDtt projects (Georgia, Michigan, and NEA ) , and that* this pattern was 
4 . \ • 

continued at the site levej. in these and in the NRCtand Florida projects, 

w^re stany of the adopted products were not even in the original knowledge 

base. Th*s transition froa validated to nonvali dated products occvurred in 

spite of the original ,RDU mandate as represented mJkhe original RF? and the 

# s 

substantial efforts made by each of the seven projects to develop a respoa**- 
* sive knowledge base.- - " • 

Three sets of issues deserve further discussion, in part to explain 
% these findings and in part to suggest future concerns for researchers and^ 
«*duca*- jonaL policy makers: * 

« thd adequacy of the pool of educational products; y 

• l&er need *for operational definitions in the design of 
programs; and 

• • * 

• the^ need t6 understand the normal process of "slippage" 
between program Tntent and program implementation. 

■ * 

the tteed for More Educational Products , 

Whatever else may be suggested by the difficulties of assembling a knowl-" 
edje^scas^, it *s clear that the pool of acceptable educational products needs 
to^be expanded in the future. On certain topics, little R&D had been done^fct/ 
the time the PDU projects were in operation* These topi^ include, career 
education (interest in which only burgeoned in the early 1970s, not allowing 
enough tine to produce a wide variety of relevant, validated products for the 
RDU program in 1976); district- or school-wide planning (which was added by 
the Seojrgaa project); inservice training** In principle^ the" last topic would 
nave been feasible had it* been limited to the few curriculum areas in which a 
Urge number of validated products existed. However, the inservice topic. was 



* There is also seme Question of how to validate educational products on 
these topics, where ogre reliance must necessarily be placed on affective 
ratings, Such as attitude scales. Additional problems confront the vali- 

n% , dation of products that only indirectly affect students— e.g, , infeervice 
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broadly applied to almost every curriculum area th&t could^be of interest 

t 

to a site, and thus stretched beyond the reservoir of available products. 

On other topics, the need for products was only recognised in the course 
of the RDU program. These included almost any curriculum area at the second^ 
*ary school level, especially on core topics such as reading. Similarly, RDU 
officials who were interviewed indicated that sites frequently sought products 
on topics outside of the basic skills— e*g*, science, language, and social 
studies—but encountered grMt difficulty in finding them. Lastly, there 
appeared to be a "heed for assistance on topics related to racial integration 
and the problems of minority groups (e.g«, products for bilingual students), 
and the available pool* of products again seemed sparse* 

In expanding the pool of educational products in the jEuture, it should , 
be -toted chat important contributions can be made from two sources* Tradi- 
tionally, one source 'has been the result of H&D efforts, in which new* Prod- 
is 

uuts are developed b> using scientifically based information in » institu- 
tion or university* department that is devoted to knowledge production ac>* 
tiviti*ii.* R&D efforts are not, however, the onlj" source of new knowledge 
tfyat 3iay bear upon the improvement of schools and schooling: there is also 
graotii,e*tase^ knowledge , which is based upon practitioners* own»experience*s. 
A /ariety of local or federal protracts have been targeted toward the knowl- 
edge development activities of practitioners* These activities can produce 
measurable* ^pr^eg^fttr7^ftd-Hr4ght-4>e c a ll ed "P&V activities (-practice and 

- * 

development)^ 

ffhatevei the source of new knowledge, however, any acceptable education 
produce oust be validated by a process which involves forma ^certification 
of impact and transportability .* Typically, the validation process involves 
a quantitative experimental or quasi-experimental evaluation, .providing 
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* Despite the fact that the use of r&d products was a major initial objective 
of the R&D program, considerable ambiguity exists concerning the meaning of 
"R&D products*. The ideas for many educational products come from ordinary 
knowledge or from the experience of practitioners* This is true both for 
so called "exemplary practice* and for many products produced by RfiD organi- 
zations. These may be excellent products, but from a "purist" perspective 
they are not research and developtnen ^ products. Prom this perspective, to 
qualify as an R6D product, the product design would have to ije based on the 
result of prior research and development or propositions in systematic theory 
preferably both. Similarly if the product design was derived from a re- 
search or theoretical basis, then i% might be considered an "R*D product" 
even though the final .development of the product lacked an, empirical vali- 
datjjpn step^ 0 
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xnfoknation about dif ferencos in test scores between groups that did and did 
not receive the 'treatment These results are then reviewed by a formally 
constituted group such as the OE/NIE Joint Dissemination Review Panel . Thus, 
<juantltdtively based validation prodtedures should probably b* viewed as <*ne 



type of minimal screen through which a proposed educational product must pass 

* - 
befote it is deemed worthy of dissemination. 

tf knowledge source and validation status are cpnsidered two indepen- 
dent iimensions, a^ two-by-two matrix may be created and is shown in Figure 3.- 
Thio matrix illustrates the proposition, that future products may come from 
R&D or 'P&D efforts, but that the validation procedure must be applied to 
products from either type of effort in order for a product to be acceptable. 

Al:>ng with the development of a larger pool of products , future efforts 
are also r*eded to identify the product characteristic*, -that facilitate im- 
plementation. For instance, interviews with project staff members and school 
pex&j.ine provided considerable anecdotal .evidence regarding what makes"' an at- 
tractive product- Such products had the advantage of: 

• A multiple adoptions in recent years, which frequently 

aeant .that there was a proximate school which could 
be visited for observational purposes* 0 

• experienced trainers, who were funded through 'federal 
. programs such as NEW to provide both pre-implementa- 

tion and follow-up service to adopting schools; and 

• well-packaged and easily available materials , usually 
provided at, or unde?, reproduction costs. 

1ft contrast, less attractive products were more likely to have been adopted 

4.n . ev places, to have had no experienced available trainers or to re<Juire 

consultant fees for training, and in some cases to have had materials that 

were not as attractively produced. Only further research can* demonstrate 

at 

the importance of these types of characteristics in contrast to those having 

• • * * 

to do* with the content/ of the product. 

* The -Need for Clearer feef initions ^ 

1 " 

A second potential lesson from the sites 1 experiences with products is 

that clearer operational definitions need to be made in the design of a pro- 

yram. Thus, we j»ted in Chapter \ that* the RFP did not describe the minimal 

validation procedure » If a product did not *cot*e from the pool of nationally^ 

validated products, what was an RDU project to do? Four of the seven projects 
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Figure 3 

* TYPES, OF EDUCATIONAL PRODUCTS BY 

— SOURCE AND VALIDATION STATUS 



■ 



r 

t * 
. » ^ 


i 

Source of Product 


» 

, Ty^e of 
1 „ Certification 


Research and 
Develoftaept 
(KiD) EffoH 

i V 


- Practice and 
H Development 
(P&D) Effort 


i Validated 

* (Formal Certi-. 
fication of 

j Intact & Trans- 
portability) 


Acceptable , 
Product 


Acceptable 
Product - / 

1 


Not Validated 


* 

Unacceptable 
Product 


# 

Unacceptable 
Product 


i 

* 


• 






relied on a review panel procedure •* During the review process, if data were 
not available on- a certain product fro? some existing test, time was needed 
for the collection of those data. To our knowledge, the EDU progzfas was not 
designed to allow for t;Ms time lag? and, if a site wished to adopt a non- 
validated product, no separate attempt was made to conduct an empirical vali- 
dation of the product's impact* Other appr oaches to product certification-- 
including judgments of face validity by experts , or judgments oft actual im- 
pact b} practitioners using non-test measures—were usedNtoo an unknown extent 
in the EDtf prograh* " • \ 

2ther operational details were also unanticipated by the designers of the 
RDU program* How Jlarge does an adequate knowledge base need to be? How often 
shoulc 1 1* be"\pdated^ What feedback is possibll so that site experiences cap 
be incorporated into the knowledge base? These and other questions might hpve 
been mor? easily resolved had there been, a't the time of the design of the RDU 
pro</raD, a national knowledge base with these operating experiences. Even now, 
th*s ^jif»wnce of a single knowledge base might reduce the problems of searching 
for and documenting accessible and acceptable products that may be included in 
local knowledge bases* At a national level, some economies of scale might be 
a=l~Jeved, and greater resources might therefore # be made .available for bith lo- 
uati..g jud screening products* However, a national "knowledge base alone, in 
the *±>s>Jve of local knowledge bases, might al£o have some disadvantages* It 
cught ->;>t be sufficiently responsive to local site needs, a concern which was* 
£xpres>sed ^ne^ HDU project staffs when asked about this issue* The typical 

fear wa^ that a' national knowledge base which was far removed from the locus 

f 

of implementation would be unable to respond in^a timely or fine-it uned manner. 

In addition, local knowledge bases would allow sites to have a fore intimate 

involvement in the product selection process, with face-to-face visits fre- 

tjuently osed to review candidate products; a national knowledge base night 

• * 

not provide the same opportunities* These advantages and disadvantages would 



* The product review panels, except in the case of the NRC, existed before 
the HDU program. Knowledge-base staff did not sutait candidate products 
to the panels in order for them to be validated for their RDU projects* 
Rathex , for thojfe projects whose states had an existing validation proce- 
dure i Pennsylvania, The NETWORK/ and Michigan), the knowledge-base staff 
considered products that had already been validated by the panels as po- 
tential knowledge-base products* 
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all have to be taken Into account in* the planning for any national knowledge 
base. , v 

Normal Shifts bet we en Intent and Implementation 

, A final set of explanations aijd implications derives from our obser-^ 
vatxon that the PDU program was designed with a purposeful tension. On 



the one hand, the goal of using existinq^RSD products can be construed as 
a technology-oriented objective* On the other hand, the RDU program^ also " 
called for sites to conduct their own needs assessments before seeing a 
product, and this can be construed -as Wuser-oriented objective. In reality, 
these two objectives can co-exist best when adequate time—usually a moderate 
period of years— is allowed for the knowledge development and needs assess&ent 
activities to inform one another. Unmet site needs, for "instance, should lead 
to the commissioning of new product development. Because the RDU program 
was not a l*ong-term program, the tension between^he two "objective* had to be 
settled in other ways, and our belief is that these compromises explain the 
transition from validated to nonvalidated products. ^ Moreover, such shifts 
between intent and implementation can occur throughout the life of * federal 
program. 

Federal programs intended to affect local practices nay be *a;d to havfe 
five phases (not necessarily occurring in simple linear order)** fc 

• federal policy .development ^ 

• federal program development 

• project design ( 

• site or service delivery development 

• practice implementation fL, 

y & s 
Figure 4 shows how these phag^g may, be depicted with regard to the RDU pro- 
gram. ■ Th'is model suggests tjiat implementation at one phase frequently be- 
comes a major component of Jthe set of plans or intentions at the next phase. 
Intent at each phase is usually embodied in a set of goals or objectives* 
As a result of either conscious or serendipitous choices in £he implemen- 
tation of each phase, activities are set in motion that are believed to ad- 
dress the goals that were developed. However, in most implementation efforts. 



* There is considerable literature that provides useful background informa- 
tion for ttjia topic. * Por exacple, see Berman, 1978; Farrar et al., 1979* 
Hage and Aiken, 1970; Louis, ,1980; McLaughlin and Berman, 1975j and March 
an4 014en, 1976. The term "slippage" was originally "used by Philip D. 

. Selznik (scp Corvin, forthcoming). 
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Figure 4 



FIVE PHASES IN SHE IMPLEMENTATION Op 
A^n-DERAL PROGRAM INTENDED TO AFFECT LOCAL PRACTICE 

_i : 



POLICY DEVELOPMENT 
PHASE 



Intent 

* \ v : 

Congressional legislation 
Federal regulations 



Implementation 

Design of RDU program, pfi- 
marily reflected in RFP 



PROGRAM DEVELOPMENT 
PHASE 



Intent 

Design of RDU program by 
NIE , primarily reflected 
in RFP 



Implementation 



Design of. seven project awards, 
based on project proposals 
Monitoring of awards 



i 



PROJECT DESIGN 
PHASE 



Intent 



Design of individual project, 
based on project pcoposal 
NIE monitoring of award 



Implementation 

Actual project organization 
Actual project procedures 




SITE DEVELOPMENT 

PHASE „S 



Intent 
* — ■ 

Actual project organization 
Actual project procedures 



Implementation 



- Adoption of specific idea {R&D 
product) for implementation 



PRACTICE; IMPLEMENTATION 
PHASE 



Intent 



Adoption of specific idea*<R*D 
product) for implementation \ 



4 U Implementation 
- Implementation of new practice 
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many Choices must beinade, both quickly and m the absence (of complete and 
sound information, ajbout^ the consequences that the choices may i}*vfe for 
meeting the objectives twrkswere intended. This basi^s^henocnenon of decision 
making ui\der conditions of uncertainty has long been acknowledged by students 
of public administration (Lmdbloom, 1959; March and Olsen/H976; Pressman 
and ttxldavsky, 1973; Bailey an<i*M6sher, 1968? Murphy, 1971). The incremen- 
tal decisions made by administrators during program implementation result 
in subtle but concrete changes in the program-- changes that, over time, 
often involve substantial evolution away from the original design and intent. 

/ a 

Ajnoncf tffe many terms used to describe this process, that of 'mutual adaptaton" 

(Beraan and McLaughlin, 1977 J has appealed rao«Cbroadly to educational re- 

searchers and policy makers. This term is, however, frequently applied with 
# 

•japxecision, an<J it is -important to define the process^ of incremental change 

*- * 
Pore concretely. < * - %. 

V.' ■ ' 

According to Corwin U$80) there are four types of adsustment between 

che projec^-a^-intended and the "real world.'" These ,/aay be classified ac- 
cording to: * 

4 ' 

• the type of decision (planned or deliberate, versus 
unplanned or inadvertant); and 

• the *type j>€ motivation for tne decision (to overcome^ 
^obstacles in implementation versus the, need to correct 

for inadequacies or ambiguities in program design). 

A cross tabulation of these produces four types, which are shown in Figure 5. 

Mutual .adjustment 'refers to ' the outcomes of decisions that involve de- 
lii^rate consideration of the changes to be made. This planned -change will 
ijWQlye some alterations in the program, and some expectati6n of changes in 
the setting in which it is implemented (Se'naan and McLaughlin, 1977). Using, 
the matrix in Figure 5, we find two sub 2 - types- of mutual- adaptation: 

s 

• Revision . Tfcis occurs as 'the program is tested in the field, ^ # 
and problems 6f implementing it a^ planned are encountered. 
-Decisi9ng are then -made about now* best to approximate the 

# * original intent under conditions of reaiity, and how to adjust 

the local setting to accept the intent. 



Specification *' Program designs are usually skeletal, and must 
be fleshed out during implementation. Policies are *ontjtnuously 
developed to reflect accumulating information about how, the pro- 



» gram operates , 



Figure 5 

Types of Change 3etween Intent and Inpletaentation 
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The outcomes of inadvertant or unplanried decision making, on the other 
hand, may be better thought of as slippage * In the case of slippage, ^ we nay 
find changes botJ> in the program and the setting, but we also find instances 
in which only the program changes. Slippage nay be good or *bad (although 
from the ration*! administrative perspective, it is clearly preferable to - 
make planful decisions where possible), but it is also unavoidable as the ~ 
press of responsibilities and tne need to find rapid closure for day-to-day 
problems occurs* ^ere are two sub-typeg of madvertant decision making: 

* Accomodation * Accomodation occurs due to administrative re- 
sistance or demands from the client or other actors in the en-** . 
vironment. Whefl and where there are "exceptions to the rule," 

* or where^ecisions are made to altejr nfles in order to achieve 
a valued opal more quickly, the program has a tendency to be 
altered more than the client or environment . 

• Mutation * Unpredictable local interpretations of the original — 
program design art fre(fiently encountered in the -implementation 
of new programs. These usually occur becau^g the implementing 
individual (s) did not fully understand what the designers wanted, ' 
but also^occur as a consequence of localized M msights, M which 

^ involve the generation of new goals or strategies which seem, to 
the imptenentor, to add to the original design. * 

The development of the design and policies govermng*the use of "knowl- 
edge" in the R&D Utilization program gave evidence of several types of change 
througn, incremental decision making. TKfe changes that occurred in the policy 
**nd program development phases stay, according to Corwjin's ll^auj analysis, 
be/t be classified as ftutual adaptation , because they revolved primarily 
around well-p±anfted changes in design to reflect the absence of .nationally 
developed product pools that net the desires of NIB to ,t£St an R&D Utiliza- 
tion strategy. The .outcomes of the process of change that has been docu- 
mented in this report, on the o£her hand, can be considered to be slippage , 
both through accomodation to the absence of appropriate products for the 
data basas and client demands, and mutation , as several of the projects spon- 
taneously converted t4 a greater emphasis upon the dissemination of improved 
problem Solving capabilities, as opposed to R&D* 

Tji the RDU^ program the "slippage" from validated to nonvalidated prod- m 
ucts mainly occurred at two phases of the fiv.e-ph4«e model* During the 
project design phase, the intended activity was the development of a knowl- 
edge base (or pool) of validated products* Chapter II described how this 



function was conducted by each of the seven projects and shoved how this 
involved accomodation in each project, to available resources. At the site 
development phase , the pertinent activity was the matching process, whereby 
sAes had to matc^\their needs to $5ie products available before making a 
fintfl adoption. This activity again led to considerable accomodation and. 
was described fully in Chapter XXX* 

Ve believe tfcat ^slippage can be attributed to the tension between the 
technology-oriented and user-oriented objectives of the RDU program* Xn 
^je original RFP, the technology-oriented elements of the program were 
dominant. Portions of the RFP clearly indicated that improvements in the 
problem-solving process used by schools were also desirable, but these as- 
pects were* not as prominent as the implementation of R&D products . During 
implementation, the balance between the technology-oriented and" use«f- 
orijggited elements o'f the program shifted, such .t&at by the t^rne the RDU 
program had been in operation for a year-and-a-half , it; was clear that some 
projects were emphasizing improvement in a .local problem-solving capacity 
luser-oriented objective) over the use of F&O products as their major 
accomplishment* By t*je end of two years, there was apparent consensus among 
the seven project didctors on the primacy of the user-oriented objective*-* 

To .summarize, the transformation of the original intent into a^iew set 
>f intents is a pattern which characterizes many organizational anH program 
uAatOt+as. Such changes/ deserve t*r> b*» the focus of further research, espe- 
nilly in light of the following observation. Previous research ok federal* 
local* implementation** has repeatedly suggested that little beneficial influ^ 
ence can be exerted by federal policies at £he iy.fth phase m our mod^l^ 
what *e have called the practice .implementation phase.*** The outcomes of 
local implementation appear to be more a function of local conditions, such 
a^ the locjll agency organization, rules of governance , and personnel . To£ 
this reason r the existing research has shown little, if any, cross-site cor- 
relation between the availability of federal resources and local implementa- 
tion outcomes (Yin et al., 1977; Benaan and McLaughlin, 1978; Miles, Fullan, 
Taylor, 1979) . This weak relationship has continually thwarted federal 



* This conclusion is based upoa observations of a meeting of the seven t 
project directors and several NIE staff members held in October 1978. 

** For example, see Yin et a^., 1977; Lambright and Fjynn, 1977; Berman and 
McLaughlin, 1978; and Ros/snblu&and XAuis* 1979. 

***?or a discussion of the disincentives of federal *p° lic i es at this stage, 
* see Sieber, 1979. . f „ fc ) 



'agencxes whose mission is to crcate^change and thereby improve services at 
the local level. Federal policies may, however, significantly influence 
implementation activities at the other four phases* in our model/ which in 
turn can constrain or enhance local adoption behavior;. For those who sup-* 
port the federal agenda < the assumption is that services might then be more 
"^"Jikely to change, even ^moderate adaptation takes place during the fifth 
phase- For those who support the local agenda, knowledge about the slip- 
page and mutual adaptation process could help to alter plans at one of the 
earlie^ four phases. Thus, any new information on changes between intent 
tQ -jspleaentation can potentially yield policy-relevant recommendations for 
both federal and ^local agencies. 





Summary +■ 

The R&D Utilization (RDP) progtam was designed by NIE^fco improve school 
practices* Such improvements were to occur, in part, as a result of the 
adoption and implementation of specific R&D "products"— i.e. , ideas curricu- 
lum changes based on previous research and testing. Thus, the organization 
of the RDU program involved the presentation, to each participating schools 
of an array of potential products for review and selection Yf the school 
staff. * 

This report has described the administrative proceSp-Aereby; 

• information about existing products was accumulated into 
.* a "knowledgebase;" 

• the materials in the knowledge base were ma£e available 
to school sites; i \ 

• the school sites were assisted in selecting the product ^ 
that most sua ted their needs; and 

• specific products were finally ^adopted by each school. > 

h major issue throughout this process had to do witn the desire to-tiave v 

sc*w^Is adopt only products that had been previously validated through em* 

pis-ical testing. Thus* the intent of the RDU program was to assure some 

quality control 'over the new ideas that would be implemented within -any 

giver School* . c ? 

* « 

In reviewing. each of the major steps in the process, this report has 
-ientified difficulties that were encountered ip fulfilling the program's 
mandate. The analysis shows that, in spite of the 'substantial efforts made 
to assure quality control, many school sitesTliltimately adopted products 
that had not been validated for their effectiveness or transportability 
through prxpr testing • In fact, up tp 50 or 60 percent of the final adop- 
tions nay have involved nonvalidated products. 

the above analysis points to .a. number of implications for the,$Edivi^- 
ual who must develop and manage pools of educational products. Chfe generic 
lesson is that the design and implementation of quality control procedures , 
for dissemination should be regarded as a two-stage process* The first^ 
stage centers" on determining which 1 information bits, programs, or other 
knowledge should be included in the pool, and the second occurs as the. cli- 
ent decides what information to use* A dissemination program can influence 
both steps, but different procedures must be designed for each stage. 



Second, the experiences of the RDU projects suggest that the iaplemen- - 
tation pf strict quality control procedures is extreaely difficult where the 
dissemination program seeks to maximize responsiveness to client needs* The 
higher the client orientation of thje dissemination progran, the more impor- 
tant it is that backup screening procedure^ be developed for use when clients 
^require information that cannot be found within the more rigorously screened^ 
pool- Without such backup procedures, the project staff may lose all control' 
over quality at various crucial stages of its relationship with a client 
school or district. 

Pinal ly, while- the attention devoted to quality control was appropriate 
in the RDU program, given its research focus, we believe that ongoing proj- 
( ects would be well advised to utilize ©ore flexible criteria for quality con- 
trol. Thus, for example, a project might develop a three-tiered ranking sys- 

* 

tern for all of its "knowledge * p Where ^ nee$ could be met through research- 
based, validated products, these would be given highest priority. However, 

products that had been field tested {but not validated) and products that 

* * 

siaspiy stet criteria of^sfcrong fa-e validity would also be available through 
^he retrieval system* There' is soae evidence from o£her analysis in our 
«jiy that quality control is important,* there is also evidence that many 
schools might have 'becccae discoursed aid night have abandoned the program 
improvement effort entirely if ^Jiey had been for«d to <&oose only fr^a a 
'liaited knowledge pool- ' 

The results also suggest several potential concerns for future educa- % 
tional policymaking* Pirst, the existing pool of acceptable products may 
not have been large enough* at the outset, to serve the varied needs of the 
sites. In socxe curriculum areas, 'such as career educath>n and other fopics 
outside of the basic skills, there were simply too few products frcw which 
to choose* A future need nay therefore be the development of a larger and % 
broader array of R&D products to match the types of problems frequently 

encountered by schools* Second/ clearer operational guidance may be needed 

m N 

for establishing an appropriate knowledge base* Among the key steps* is the 

implementation *of specific validation procedures to certify new products that 



Por example, whothe^or not an adopted product has been* field tested 
strongly predicts the degree to which it affects the school aad its 
pupils and staff (Louis, 1980)* 



may emerge in the course of an educational change program. InNphe case of 
RDU, such procedures we,re called fdr and ased, but the design ot the pro- 
gram*, did not allow for the time lag needed ^o accumulate new empirical 
evidence before a school could proceed with Adoption* Third, v the RDU ex* 
penence raises some general questions about the lively slippage that will 
occur, from intent to implementation, throughout all phases Qf a federally- 
designed program. Future programs should make a better attempt to antici- 
pate this slippage process* 

• Overall, these cycerns m implementing school change through federal 
intervention are impo rt ant not only to the RDU program but also to other 
federal programs in education . The issues raised should therefore be com- 
parei ^ith the lessons learned froci other federal programs, in the hope of 
improving the design of future programs. 
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